Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250626 14:04:16.183921 1071 test_util.cc:276] Using random seed: 158754397
2025-06-26T14:04:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T14:04:16Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskmpJjah/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskmpJjah/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/security.properties
2025-06-26 14:04:22: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskmpJjah/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskmpJjah/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250626 14:04:30.668844 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.254:36707
--webserver_interface=127.1.11.254
--webserver_port=0
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250626 14:04:30.771067 1313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:30.771302 1313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:30.771342 1313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:04:30.774557 1313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:04:30.774622 1313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:30.774644 1313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:04:30.774662 1313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:04:30.778465 1313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.254:36707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.11.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:30.779554 1313 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:30.780687 1313 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:04:30.787587 1319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:30.787645 1318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:30.787580 1321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:30.787885 1313 server_base.cc:1048] running on GCE node
I20250626 14:04:30.788401 1313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:30.789323 1313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:30.790467 1313 hybrid_clock.cc:648] HybridClock initialized: now 1750946670790450 us; error 37 us; skew 500 ppm
I20250626 14:04:30.792271 1313 webserver.cc:469] Webserver started at http://127.1.11.254:45241/ using document root <none> and password file <none>
I20250626 14:04:30.792874 1313 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:30.792930 1313 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:30.793112 1313 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 14:04:30.794672 1313 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/instance:
uuid: "24ad911d71b6433f8c7939a1d1f7b782"
format_stamp: "Formatted at 2025-06-26 14:04:30 on dist-test-slave-rptj"
I20250626 14:04:30.795128 1313 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal/instance:
uuid: "24ad911d71b6433f8c7939a1d1f7b782"
format_stamp: "Formatted at 2025-06-26 14:04:30 on dist-test-slave-rptj"
I20250626 14:04:30.798473 1313 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250626 14:04:30.800638 1327 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:30.801632 1313 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:30.801735 1313 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
uuid: "24ad911d71b6433f8c7939a1d1f7b782"
format_stamp: "Formatted at 2025-06-26 14:04:30 on dist-test-slave-rptj"
I20250626 14:04:30.801821 1313 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:04:30.834548 1313 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:30.835220 1313 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:30.835399 1313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:30.842280 1313 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.254:36707
I20250626 14:04:30.842289 1379 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.254:36707 every 8 connection(s)
I20250626 14:04:30.843282 1313 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/info.pb
I20250626 14:04:30.844475 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 1313
I20250626 14:04:30.844633 1071 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal/instance
I20250626 14:04:30.845944 1380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:30.845948 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.253:44447
--webserver_interface=127.1.11.253
--webserver_port=0
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250626 14:04:30.849431 1380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } has no permanent_uuid. Determining permanent_uuid...
I20250626 14:04:30.860402 1380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } has no permanent_uuid. Determining permanent_uuid...
W20250626 14:04:30.861198 1330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:04:30.862035 1380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:30.911252 1380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } attempt: 1
W20250626 14:04:30.912731 1380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:04:30.951241 1381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:30.951509 1381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:30.951586 1381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:04:30.954834 1381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:04:30.954905 1381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:30.954942 1381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:04:30.954972 1381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:04:30.958731 1381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.253:44447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.11.253
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:30.959880 1381 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:30.961050 1381 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:04:30.968281 1392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:30.968266 1389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:30.968266 1390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:30.968677 1381 server_base.cc:1048] running on GCE node
I20250626 14:04:30.969348 1381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:30.970330 1381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:30.971521 1381 hybrid_clock.cc:648] HybridClock initialized: now 1750946670971499 us; error 35 us; skew 500 ppm
I20250626 14:04:30.973361 1381 webserver.cc:469] Webserver started at http://127.1.11.253:37937/ using document root <none> and password file <none>
I20250626 14:04:30.973932 1381 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:30.973990 1381 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:30.974202 1381 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 14:04:30.977259 1381 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/instance:
uuid: "07532b7648e4414789ed2b099bad2b19"
format_stamp: "Formatted at 2025-06-26 14:04:30 on dist-test-slave-rptj"
I20250626 14:04:30.977777 1381 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal/instance:
uuid: "07532b7648e4414789ed2b099bad2b19"
format_stamp: "Formatted at 2025-06-26 14:04:30 on dist-test-slave-rptj"
I20250626 14:04:30.980880 1380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } attempt: 2
I20250626 14:04:30.981418 1381 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
W20250626 14:04:30.982220 1380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:30.983875 1398 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:30.984959 1381 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250626 14:04:30.985062 1381 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
uuid: "07532b7648e4414789ed2b099bad2b19"
format_stamp: "Formatted at 2025-06-26 14:04:30 on dist-test-slave-rptj"
I20250626 14:04:30.985154 1381 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:04:31.001621 1381 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:31.002280 1381 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:31.002447 1381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:31.009227 1381 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.253:44447
I20250626 14:04:31.009234 1450 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.253:44447 every 8 connection(s)
I20250626 14:04:31.010242 1381 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
I20250626 14:04:31.011214 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 1381
I20250626 14:04:31.011319 1071 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal/instance
I20250626 14:04:31.012113 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.252:37925
--webserver_interface=127.1.11.252
--webserver_port=0
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250626 14:04:31.013422 1451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:31.016990 1451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } has no permanent_uuid. Determining permanent_uuid...
I20250626 14:04:31.025714 1451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } has no permanent_uuid. Determining permanent_uuid...
I20250626 14:04:31.031646 1451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } has no permanent_uuid. Determining permanent_uuid...
W20250626 14:04:31.032377 1399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:04:31.033157 1451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.11.252:37925: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:04:31.073396 1380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } attempt: 3
I20250626 14:04:31.077593 1380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } has no permanent_uuid. Determining permanent_uuid...
W20250626 14:04:31.078711 1380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.11.252:37925: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:04:31.082341 1451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } attempt: 1
W20250626 14:04:31.083498 1451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.11.252:37925: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:04:31.109975 1380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } attempt: 1
W20250626 14:04:31.111323 1380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.11.252:37925: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:04:31.120154 1452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:31.120420 1452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:31.120464 1452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:04:31.123819 1452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:04:31.123931 1452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:31.123968 1452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:04:31.124001 1452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:04:31.127939 1452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.252:37925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.11.252
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:31.129130 1452 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:31.130285 1452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:04:31.137106 1463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:31.137072 1461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:31.137176 1452 server_base.cc:1048] running on GCE node
W20250626 14:04:31.137072 1460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:31.137831 1452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:31.138760 1452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:31.139935 1452 hybrid_clock.cc:648] HybridClock initialized: now 1750946671139911 us; error 36 us; skew 500 ppm
I20250626 14:04:31.142022 1452 webserver.cc:469] Webserver started at http://127.1.11.252:43659/ using document root <none> and password file <none>
I20250626 14:04:31.142611 1452 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:31.142678 1452 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:31.142863 1452 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 14:04:31.144518 1452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/instance:
uuid: "81db61eed5584e7e8b0cf40e5441f88e"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:31.145040 1452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal/instance:
uuid: "81db61eed5584e7e8b0cf40e5441f88e"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:31.148665 1452 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20250626 14:04:31.150990 1469 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:31.151672 1451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } attempt: 2
I20250626 14:04:31.152154 1452 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:31.152280 1452 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
uuid: "81db61eed5584e7e8b0cf40e5441f88e"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:31.152429 1452 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250626 14:04:31.153048 1451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.11.252:37925: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:04:31.168009 1452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:31.168766 1452 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:31.168932 1452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:31.176314 1452 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.252:37925
I20250626 14:04:31.176328 1521 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.252:37925 every 8 connection(s)
I20250626 14:04:31.177398 1452 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/info.pb
I20250626 14:04:31.177785 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 1452
I20250626 14:04:31.177896 1071 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal/instance
I20250626 14:04:31.180035 1522 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:31.184649 1522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } has no permanent_uuid. Determining permanent_uuid...
I20250626 14:04:31.186724 1380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } attempt: 2
I20250626 14:04:31.194058 1522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } has no permanent_uuid. Determining permanent_uuid...
I20250626 14:04:31.195854 1380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Bootstrap starting.
I20250626 14:04:31.198450 1380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:31.199357 1380 log.cc:826] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Log is configured to *not* fsync() on all Append() calls
I20250626 14:04:31.201654 1380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: No bootstrap required, opened a new log
I20250626 14:04:31.203173 1522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } has no permanent_uuid. Determining permanent_uuid...
I20250626 14:04:31.205372 1380 raft_consensus.cc:357] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:31.205569 1380 raft_consensus.cc:383] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:31.205648 1380 raft_consensus.cc:738] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ad911d71b6433f8c7939a1d1f7b782, State: Initialized, Role: FOLLOWER
I20250626 14:04:31.206223 1380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:31.206956 1529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:31.207087 1529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:31.207654 1380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:04:31.209713 1522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Bootstrap starting.
I20250626 14:04:31.212657 1522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:31.213586 1522 log.cc:826] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Log is configured to *not* fsync() on all Append() calls
W20250626 14:04:31.214450 1543 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 14:04:31.214526 1543 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 14:04:31.215487 1522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: No bootstrap required, opened a new log
I20250626 14:04:31.214404 1313 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: OK, instance_id { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" instance_seqno: 1750946671007313 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250626 14:04:31.216784 1313 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:04:31.218513 1522 raft_consensus.cc:357] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:31.218719 1522 raft_consensus.cc:383] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:31.218783 1522 raft_consensus.cc:738] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81db61eed5584e7e8b0cf40e5441f88e, State: Initialized, Role: FOLLOWER
I20250626 14:04:31.219208 1522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:31.219681 1547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:31.219820 1547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:31.220480 1522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: configured and running, proceeding with master startup.
W20250626 14:04:31.227732 1561 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 14:04:31.227870 1561 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 14:04:31.228804 1452 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: OK, instance_id { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" instance_seqno: 1750946671007313 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250626 14:04:31.231161 1452 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:04:31.244194 1451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } attempt: 3
I20250626 14:04:31.249382 1547 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:31.249585 1547 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:31.250679 1547 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
I20250626 14:04:31.251603 1451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Bootstrap starting.
I20250626 14:04:31.252741 1354 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81db61eed5584e7e8b0cf40e5441f88e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ad911d71b6433f8c7939a1d1f7b782" is_pre_election: true
I20250626 14:04:31.253064 1354 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81db61eed5584e7e8b0cf40e5441f88e in term 0.
I20250626 14:04:31.253790 1470 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782, 81db61eed5584e7e8b0cf40e5441f88e; no voters:
I20250626 14:04:31.254139 1547 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:31.254220 1547 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:31.254268 1547 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:31.255493 1547 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:31.255911 1547 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 1 election: Requested vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
I20250626 14:04:31.256691 1354 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81db61eed5584e7e8b0cf40e5441f88e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ad911d71b6433f8c7939a1d1f7b782"
I20250626 14:04:31.256824 1354 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:31.258085 1354 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81db61eed5584e7e8b0cf40e5441f88e in term 1.
I20250626 14:04:31.258402 1470 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782, 81db61eed5584e7e8b0cf40e5441f88e; no voters:
I20250626 14:04:31.258615 1547 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:31.258862 1547 raft_consensus.cc:695] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 1 LEADER]: Becoming Leader. State: Replica: 81db61eed5584e7e8b0cf40e5441f88e, State: Running, Role: LEADER
I20250626 14:04:31.259207 1547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:31.262216 1451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:31.262776 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81db61eed5584e7e8b0cf40e5441f88e. Latest consensus state: current_term: 1 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:31.262972 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:31.264132 1451 log.cc:826] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Log is configured to *not* fsync() on all Append() calls
I20250626 14:04:31.269881 1451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: No bootstrap required, opened a new log
I20250626 14:04:31.274128 1451 raft_consensus.cc:357] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:31.274353 1451 raft_consensus.cc:383] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:31.274415 1451 raft_consensus.cc:738] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07532b7648e4414789ed2b099bad2b19, State: Initialized, Role: FOLLOWER
I20250626 14:04:31.274997 1451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:31.275574 1568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:31.275851 1568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:31.276372 1451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:04:31.284634 1566 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 14:04:31.286010 1566 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 14:04:31.276398 1424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81db61eed5584e7e8b0cf40e5441f88e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07532b7648e4414789ed2b099bad2b19"
I20250626 14:04:31.288796 1424 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:31.290189 1424 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81db61eed5584e7e8b0cf40e5441f88e in term 1.
I20250626 14:04:31.276515 1425 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81db61eed5584e7e8b0cf40e5441f88e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07532b7648e4414789ed2b099bad2b19" is_pre_election: true
I20250626 14:04:31.290584 1425 raft_consensus.cc:2374] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 81db61eed5584e7e8b0cf40e5441f88e in term 1. Re-sending same reply.
I20250626 14:04:31.296417 1425 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 1 FOLLOWER]: Refusing update from remote peer 81db61eed5584e7e8b0cf40e5441f88e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 14:04:31.297271 1564 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [LEADER]: Connected to new peer: Peer: permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:31.301594 1568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81db61eed5584e7e8b0cf40e5441f88e. Latest consensus state: current_term: 1 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:31.301724 1568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:31.307780 1568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:31.307894 1568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:31.313124 1580 mvcc.cc:204] Tried to move back new op lower bound from 7171877565620137984 to 7171877565481463808. Current Snapshot: MvccSnapshot[applied={T|T < 7171877565620137984}]
W20250626 14:04:31.314807 1585 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 14:04:31.314992 1585 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 14:04:31.318761 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:31.318917 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:31.319427 1354 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 1 FOLLOWER]: Refusing update from remote peer 81db61eed5584e7e8b0cf40e5441f88e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 14:04:31.320091 1569 mvcc.cc:204] Tried to move back new op lower bound from 7171877565620137984 to 7171877565481463808. Current Snapshot: MvccSnapshot[applied={T|T < 7171877565620137984}]
I20250626 14:04:31.320366 1547 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:31.323889 1566 catalog_manager.cc:1349] Generated new cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:31.323967 1566 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 14:04:31.330659 1529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81db61eed5584e7e8b0cf40e5441f88e. Latest consensus state: current_term: 1 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:31.330821 1529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:31.330922 1529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:31.331032 1529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: This master's current role is: FOLLOWER
W20250626 14:04:31.336776 1333 tablet.cc:2378] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 14:04:31.350636 1589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:31.350800 1589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:31.354147 1590 mvcc.cc:204] Tried to move back new op lower bound from 7171877565620137984 to 7171877565481463808. Current Snapshot: MvccSnapshot[applied={T|T < 7171877565620137984}]
I20250626 14:04:31.361150 1566 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 14:04:31.362416 1566 catalog_manager.cc:1506] Loading token signing keys...
I20250626 14:04:31.383363 1566 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Generated new TSK 0
I20250626 14:04:31.384407 1566 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 14:04:31.385102 1566 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250626 14:04:31.385604 1566 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250626 14:04:31.411286 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.11.193:0
--local_ip_for_outbound_sockets=127.1.11.193
--webserver_interface=127.1.11.193
--webserver_port=0
--tserver_master_addrs=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250626 14:04:31.576506 1596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:31.576853 1596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:31.576946 1596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:04:31.581470 1596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:31.581661 1596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.11.193
I20250626 14:04:31.586472 1596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.11.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.11.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--never_fsync=true
--heap_profile_path=/tmp/kudu.1596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.11.193
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:31.587680 1596 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:31.588927 1596 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:04:31.597246 1604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:31.597368 1602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:31.597251 1601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:31.597676 1596 server_base.cc:1048] running on GCE node
I20250626 14:04:31.598079 1596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:31.598667 1596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:31.599833 1596 hybrid_clock.cc:648] HybridClock initialized: now 1750946671599814 us; error 33 us; skew 500 ppm
I20250626 14:04:31.601812 1596 webserver.cc:469] Webserver started at http://127.1.11.193:43113/ using document root <none> and password file <none>
I20250626 14:04:31.602375 1596 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:31.602432 1596 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:31.602619 1596 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 14:04:31.604218 1596 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/data/instance:
uuid: "4e57adb83e9f451488ed1d0034043e7b"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:31.604720 1596 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/instance:
uuid: "4e57adb83e9f451488ed1d0034043e7b"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:31.608093 1596 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20250626 14:04:31.610344 1610 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:31.611415 1596 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250626 14:04:31.611524 1596 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal
uuid: "4e57adb83e9f451488ed1d0034043e7b"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:31.611621 1596 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:04:31.628186 1596 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:31.628947 1596 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:31.629124 1596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:31.629686 1596 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 14:04:31.630669 1596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 14:04:31.630726 1596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:31.630767 1596 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 14:04:31.630807 1596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:31.640028 1596 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.193:35671
I20250626 14:04:31.640064 1723 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.193:35671 every 8 connection(s)
I20250626 14:04:31.641161 1596 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/data/info.pb
I20250626 14:04:31.647318 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 1596
I20250626 14:04:31.647444 1071 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/instance
I20250626 14:04:31.650676 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.11.194:0
--local_ip_for_outbound_sockets=127.1.11.194
--webserver_interface=127.1.11.194
--webserver_port=0
--tserver_master_addrs=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250626 14:04:31.651710 1725 heartbeater.cc:344] Connected to a master server at 127.1.11.254:36707
I20250626 14:04:31.651798 1724 heartbeater.cc:344] Connected to a master server at 127.1.11.252:37925
I20250626 14:04:31.651850 1726 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:31.651979 1726 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:31.651978 1725 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:31.651978 1724 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:31.652654 1725 heartbeater.cc:507] Master 127.1.11.254:36707 requested a full tablet report, sending...
I20250626 14:04:31.652835 1724 heartbeater.cc:507] Master 127.1.11.252:37925 requested a full tablet report, sending...
I20250626 14:04:31.652652 1726 heartbeater.cc:507] Master 127.1.11.253:44447 requested a full tablet report, sending...
I20250626 14:04:31.654147 1485 ts_manager.cc:194] Registered new tserver with Master: 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:31.654147 1344 ts_manager.cc:194] Registered new tserver with Master: 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:31.654147 1414 ts_manager.cc:194] Registered new tserver with Master: 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:31.656059 1485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.11.193:50791
W20250626 14:04:31.760596 1731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:31.760865 1731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:31.760927 1731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:04:31.764091 1731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:31.764237 1731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.11.194
I20250626 14:04:31.768105 1731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.11.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.11.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--never_fsync=true
--heap_profile_path=/tmp/kudu.1731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.11.194
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:31.769309 1731 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:31.770485 1731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:04:31.778637 1737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:31.778636 1736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:31.778728 1739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:31.779009 1731 server_base.cc:1048] running on GCE node
I20250626 14:04:31.779417 1731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:31.780045 1731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:31.781189 1731 hybrid_clock.cc:648] HybridClock initialized: now 1750946671781175 us; error 26 us; skew 500 ppm
I20250626 14:04:31.783105 1731 webserver.cc:469] Webserver started at http://127.1.11.194:38589/ using document root <none> and password file <none>
I20250626 14:04:31.783669 1731 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:31.783730 1731 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:31.783932 1731 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 14:04:31.785604 1731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/data/instance:
uuid: "e517893e13f546c0b40d976633b49651"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:31.786077 1731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/instance:
uuid: "e517893e13f546c0b40d976633b49651"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:31.789534 1731 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250626 14:04:31.791677 1745 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:31.792830 1731 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250626 14:04:31.792943 1731 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal
uuid: "e517893e13f546c0b40d976633b49651"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:31.793035 1731 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:04:31.812314 1731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:31.813035 1731 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:31.813210 1731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:31.813769 1731 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 14:04:31.814714 1731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 14:04:31.814765 1731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:31.814821 1731 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 14:04:31.814846 1731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:31.823699 1731 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.194:44397
I20250626 14:04:31.823724 1858 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.194:44397 every 8 connection(s)
I20250626 14:04:31.824765 1731 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/data/info.pb
I20250626 14:04:31.828137 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 1731
I20250626 14:04:31.828251 1071 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/instance
I20250626 14:04:31.830719 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.11.195:0
--local_ip_for_outbound_sockets=127.1.11.195
--webserver_interface=127.1.11.195
--webserver_port=0
--tserver_master_addrs=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250626 14:04:31.835664 1859 heartbeater.cc:344] Connected to a master server at 127.1.11.252:37925
I20250626 14:04:31.835724 1860 heartbeater.cc:344] Connected to a master server at 127.1.11.254:36707
I20250626 14:04:31.835952 1859 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:31.835952 1860 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:31.836678 1859 heartbeater.cc:507] Master 127.1.11.252:37925 requested a full tablet report, sending...
I20250626 14:04:31.836678 1860 heartbeater.cc:507] Master 127.1.11.254:36707 requested a full tablet report, sending...
I20250626 14:04:31.837287 1861 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:31.837368 1861 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:31.837571 1861 heartbeater.cc:507] Master 127.1.11.253:44447 requested a full tablet report, sending...
I20250626 14:04:31.837908 1344 ts_manager.cc:194] Registered new tserver with Master: e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:31.837906 1485 ts_manager.cc:194] Registered new tserver with Master: e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:31.838285 1414 ts_manager.cc:194] Registered new tserver with Master: e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:31.838455 1485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.11.194:56877
W20250626 14:04:31.947402 1866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:31.947640 1866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:31.947695 1866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:04:31.950896 1866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:31.951017 1866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.11.195
I20250626 14:04:31.954761 1866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.11.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.11.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--never_fsync=true
--heap_profile_path=/tmp/kudu.1866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.11.195
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:31.955850 1866 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:31.957031 1866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:04:31.964623 1874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:31.964756 1872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:31.964629 1871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:31.964865 1866 server_base.cc:1048] running on GCE node
I20250626 14:04:31.965312 1866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:31.965899 1866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:31.967046 1866 hybrid_clock.cc:648] HybridClock initialized: now 1750946671967032 us; error 27 us; skew 500 ppm
I20250626 14:04:31.969369 1866 webserver.cc:469] Webserver started at http://127.1.11.195:46329/ using document root <none> and password file <none>
I20250626 14:04:31.970089 1866 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:31.970173 1866 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:31.970412 1866 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 14:04:31.972148 1866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/data/instance:
uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:31.972687 1866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/instance:
uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:31.976148 1866 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250626 14:04:31.978420 1880 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:31.979475 1866 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:31.979583 1866 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal
uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:31.979691 1866 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:04:31.993252 1866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:31.993973 1866 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:31.994145 1866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:31.994728 1866 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 14:04:31.995728 1866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 14:04:31.995777 1866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:31.995819 1866 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 14:04:31.995836 1866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:32.004869 1866 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.195:43231
I20250626 14:04:32.004895 1993 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.195:43231 every 8 connection(s)
I20250626 14:04:32.005854 1866 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/data/info.pb
I20250626 14:04:32.008345 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 1866
I20250626 14:04:32.008450 1071 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/instance
I20250626 14:04:32.016736 1994 heartbeater.cc:344] Connected to a master server at 127.1.11.252:37925
I20250626 14:04:32.016800 1995 heartbeater.cc:344] Connected to a master server at 127.1.11.254:36707
I20250626 14:04:32.016752 1996 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:32.017006 1995 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:32.017107 1996 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:32.017006 1994 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:32.017679 1995 heartbeater.cc:507] Master 127.1.11.254:36707 requested a full tablet report, sending...
I20250626 14:04:32.017889 1994 heartbeater.cc:507] Master 127.1.11.252:37925 requested a full tablet report, sending...
I20250626 14:04:32.017680 1996 heartbeater.cc:507] Master 127.1.11.253:44447 requested a full tablet report, sending...
I20250626 14:04:32.018843 1415 ts_manager.cc:194] Registered new tserver with Master: 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:32.018931 1485 ts_manager.cc:194] Registered new tserver with Master: 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:32.019528 1485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.11.195:54487
I20250626 14:04:32.019986 1344 ts_manager.cc:194] Registered new tserver with Master: 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:32.033643 1071 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 14:04:32.047935 1071 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmpJjah/build/debug/bin/kudu with pid 1381
I20250626 14:04:32.049836 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_68167203433a4b67aadc617167cce998"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:32.050150 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_22f5000aea364833a97948c4478f7e27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:32.050475 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_1e68626ebae94b208424f5f5a66dd06e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:32.050520 1479 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_8a07386d09424c52995940144f8d5d3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:32.050597 1477 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_72f9307d2f5247a0ae4499c953ea1650"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:32.050789 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_3a3f21b3c0e54e438561e02ab79fa36b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:32.051051 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_e6e7d4c291af4d29baf43e204c5292f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:32.051302 1481 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_fb8ebc7aa8894f28b4d79252382f356a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:32.051553 1480 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_0be8dc44c93040319885961d6e4a0608"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:32.051805 1478 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_f9b5b5e2dcf64f9e889c618242274d25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 14:04:32.093688 1484 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1e68626ebae94b208424f5f5a66dd06e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 14:04:32.093679 1477 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_72f9307d2f5247a0ae4499c953ea1650 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 14:04:32.093832 1485 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_68167203433a4b67aadc617167cce998 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 14:04:32.093679 1479 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8a07386d09424c52995940144f8d5d3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 14:04:32.156508 1726 heartbeater.cc:646] Failed to heartbeat to 127.1.11.253:44447 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:32.158334 1724 heartbeater.cc:499] Master 127.1.11.252:37925 was elected leader, sending a full tablet report...
I20250626 14:04:32.215876 1543 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:32.215953 1543 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: loading cluster ID for follower catalog manager: success
I20250626 14:04:32.218585 1543 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: acquiring CA information for follower catalog manager: success
I20250626 14:04:32.220320 1543 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 14:04:32.340049 1859 heartbeater.cc:499] Master 127.1.11.252:37925 was elected leader, sending a full tablet report...
W20250626 14:04:32.340961 1861 heartbeater.cc:646] Failed to heartbeat to 127.1.11.253:44447 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:04:32.395833 1472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:04:32.396741 1472 consensus_peers.cc:487] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e -> Peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Couldn't send request to peer 07532b7648e4414789ed2b099bad2b19. Status: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 14:04:32.521368 1994 heartbeater.cc:499] Master 127.1.11.252:37925 was elected leader, sending a full tablet report...
W20250626 14:04:32.522326 1996 heartbeater.cc:646] Failed to heartbeat to 127.1.11.253:44447 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:32.557654 1484 catalog_manager.cc:2149] created HMS catalog entry for table [id=aa8a6a6f408d454abdc5948e21f84233]
I20250626 14:04:32.566516 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_1e68626ebae94b208424f5f5a66dd06e" } new_table_name: "default.Table_afd26acd01c7422fbb06520af5a4f76c" modify_external_catalogs: true
I20250626 14:04:32.580368 1928 tablet_service.cc:1468] Processing CreateTablet for tablet 6a9e685f802d47c5b8d79c17c6ff8427 (DEFAULT_TABLE table=default.table_1e68626ebae94b208424f5f5a66dd06e [id=aa8a6a6f408d454abdc5948e21f84233]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.581442 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9e685f802d47c5b8d79c17c6ff8427. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.588971 1793 tablet_service.cc:1468] Processing CreateTablet for tablet 6a9e685f802d47c5b8d79c17c6ff8427 (DEFAULT_TABLE table=default.table_1e68626ebae94b208424f5f5a66dd06e [id=aa8a6a6f408d454abdc5948e21f84233]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.590055 1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9e685f802d47c5b8d79c17c6ff8427. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.595546 1658 tablet_service.cc:1468] Processing CreateTablet for tablet 6a9e685f802d47c5b8d79c17c6ff8427 (DEFAULT_TABLE table=default.table_1e68626ebae94b208424f5f5a66dd06e [id=aa8a6a6f408d454abdc5948e21f84233]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.596589 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9e685f802d47c5b8d79c17c6ff8427. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.598075 2059 tablet_bootstrap.cc:492] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:32.600688 2059 tablet_bootstrap.cc:654] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.601711 2059 log.cc:826] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99: Log is configured to *not* fsync() on all Append() calls
I20250626 14:04:32.602233 2060 tablet_bootstrap.cc:492] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:32.603338 2061 tablet_bootstrap.cc:492] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:32.605131 2060 tablet_bootstrap.cc:654] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.605373 2061 tablet_bootstrap.cc:654] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.606068 2060 log.cc:826] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651: Log is configured to *not* fsync() on all Append() calls
I20250626 14:04:32.606279 2061 log.cc:826] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b: Log is configured to *not* fsync() on all Append() calls
I20250626 14:04:32.607834 2061 tablet_bootstrap.cc:492] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:32.607851 2060 tablet_bootstrap.cc:492] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:32.608008 2061 ts_tablet_manager.cc:1397] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250626 14:04:32.608029 2060 ts_tablet_manager.cc:1397] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250626 14:04:32.610641 1485 catalog_manager.cc:2149] created HMS catalog entry for table [id=fa7ec8992cb04805a27ff28512572a81]
I20250626 14:04:32.611110 2061 raft_consensus.cc:357] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:32.611325 2061 raft_consensus.cc:383] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.611375 2061 raft_consensus.cc:738] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.611357 2060 raft_consensus.cc:357] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:32.611585 2060 raft_consensus.cc:383] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.611642 2060 raft_consensus.cc:738] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.613615 2059 tablet_bootstrap.cc:492] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:32.613842 2059 ts_tablet_manager.cc:1397] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.004s
I20250626 14:04:32.617214 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_68167203433a4b67aadc617167cce998" } modify_external_catalogs: true
I20250626 14:04:32.617367 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_68167203433a4b67aadc617167cce998" } modify_external_catalogs: true
I20250626 14:04:32.617542 2059 raft_consensus.cc:357] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:32.617873 2059 raft_consensus.cc:383] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.617986 2059 raft_consensus.cc:738] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.618584 2059 consensus_queue.cc:260] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:32.619670 2059 ts_tablet_manager.cc:1428] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250626 14:04:32.622277 1928 tablet_service.cc:1468] Processing CreateTablet for tablet 703110ded5c348cb96e47b437f340375 (DEFAULT_TABLE table=default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.622355 1658 tablet_service.cc:1468] Processing CreateTablet for tablet 703110ded5c348cb96e47b437f340375 (DEFAULT_TABLE table=default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.622577 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703110ded5c348cb96e47b437f340375. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.622639 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703110ded5c348cb96e47b437f340375. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.624984 1793 tablet_service.cc:1468] Processing CreateTablet for tablet 703110ded5c348cb96e47b437f340375 (DEFAULT_TABLE table=default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.625293 1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703110ded5c348cb96e47b437f340375. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.626351 2061 consensus_queue.cc:260] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:32.627435 2061 ts_tablet_manager.cc:1428] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250626 14:04:32.629268 2061 tablet_bootstrap.cc:492] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:32.630549 2061 tablet_bootstrap.cc:654] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.631657 2061 tablet_bootstrap.cc:492] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:32.631744 2061 ts_tablet_manager.cc:1397] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:32.632344 2061 raft_consensus.cc:357] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.632453 2061 raft_consensus.cc:383] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.632484 2061 raft_consensus.cc:738] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.632659 2061 consensus_queue.cc:260] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.633251 2061 ts_tablet_manager.cc:1428] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250626 14:04:32.636449 2059 tablet_bootstrap.cc:492] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:32.637838 2059 tablet_bootstrap.cc:654] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.638311 2060 consensus_queue.cc:260] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:32.639447 2060 ts_tablet_manager.cc:1428] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.031s user 0.004s sys 0.001s
I20250626 14:04:32.641217 2060 tablet_bootstrap.cc:492] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:32.642622 2060 tablet_bootstrap.cc:654] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
W20250626 14:04:32.642976 1727 tablet.cc:2378] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 14:04:32.646252 2059 tablet_bootstrap.cc:492] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:32.646354 2059 ts_tablet_manager.cc:1397] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20250626 14:04:32.646950 2059 raft_consensus.cc:357] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.647047 2059 raft_consensus.cc:383] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.647078 2059 raft_consensus.cc:738] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.647197 2059 consensus_queue.cc:260] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.647441 2059 ts_tablet_manager.cc:1428] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:32.650504 2060 tablet_bootstrap.cc:492] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:32.650606 2060 ts_tablet_manager.cc:1397] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20250626 14:04:32.651132 2060 raft_consensus.cc:357] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.651224 2060 raft_consensus.cc:383] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.651250 2060 raft_consensus.cc:738] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.651669 2060 consensus_queue.cc:260] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.651984 2060 ts_tablet_manager.cc:1428] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 14:04:32.665042 1479 catalog_manager.cc:2149] created HMS catalog entry for table [id=4f2719251ee84bc9af2e04e05aa3b738]
I20250626 14:04:32.672207 1479 master_service.cc:948] ReplaceTablet: received request to replace tablet 6a9e685f802d47c5b8d79c17c6ff8427 from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:32.679214 1658 tablet_service.cc:1468] Processing CreateTablet for tablet 34cd6172930d497698cb9239220566af (DEFAULT_TABLE table=default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.679564 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34cd6172930d497698cb9239220566af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.682035 2061 tablet_bootstrap.cc:492] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:32.683096 2061 tablet_bootstrap.cc:654] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.683511 1479 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Sending DeleteTablet for 3 replicas of tablet 6a9e685f802d47c5b8d79c17c6ff8427
I20250626 14:04:32.684151 1658 tablet_service.cc:1515] Processing DeleteTablet for tablet 6a9e685f802d47c5b8d79c17c6ff8427 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f5ce7fc87164ae1ace91557275285f8) from {username='slave'} at 127.0.0.1:43706
I20250626 14:04:32.684406 2061 tablet_bootstrap.cc:492] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:32.684480 2061 ts_tablet_manager.cc:1397] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250626 14:04:32.684782 1479 catalog_manager.cc:6652] ReplaceTablet: tablet 6a9e685f802d47c5b8d79c17c6ff8427 deleted and replaced by tablet 5f5ce7fc87164ae1ace91557275285f8
I20250626 14:04:32.684957 1793 tablet_service.cc:1468] Processing CreateTablet for tablet 34cd6172930d497698cb9239220566af (DEFAULT_TABLE table=default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.685081 2061 raft_consensus.cc:357] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.685185 2061 raft_consensus.cc:383] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.685216 2061 raft_consensus.cc:738] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.685257 1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34cd6172930d497698cb9239220566af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.685483 1927 tablet_service.cc:1515] Processing DeleteTablet for tablet 6a9e685f802d47c5b8d79c17c6ff8427 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f5ce7fc87164ae1ace91557275285f8) from {username='slave'} at 127.0.0.1:54640
I20250626 14:04:32.686124 2061 consensus_queue.cc:260] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.687541 1792 tablet_service.cc:1515] Processing DeleteTablet for tablet 6a9e685f802d47c5b8d79c17c6ff8427 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f5ce7fc87164ae1ace91557275285f8) from {username='slave'} at 127.0.0.1:48162
I20250626 14:04:32.688104 1928 tablet_service.cc:1468] Processing CreateTablet for tablet 34cd6172930d497698cb9239220566af (DEFAULT_TABLE table=default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.688371 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34cd6172930d497698cb9239220566af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.688683 2060 tablet_bootstrap.cc:492] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:32.689466 2074 tablet_replica.cc:331] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:32.689715 2060 tablet_bootstrap.cc:654] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.689950 2074 raft_consensus.cc:2241] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:32.690292 2074 raft_consensus.cc:2270] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:32.691692 2061 ts_tablet_manager.cc:1428] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250626 14:04:32.691759 1479 master_service.cc:948] ReplaceTablet: received request to replace tablet 6a9e685f802d47c5b8d79c17c6ff8427 from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:32.692018 1479 master_service.cc:948] ReplaceTablet: received request to replace tablet 6a9e685f802d47c5b8d79c17c6ff8427 from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:32.692101 2075 tablet_replica.cc:331] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:32.692250 1479 master_service.cc:948] ReplaceTablet: received request to replace tablet 6a9e685f802d47c5b8d79c17c6ff8427 from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:32.692443 2075 raft_consensus.cc:2241] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:32.692734 2075 raft_consensus.cc:2270] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:32.693987 2075 ts_tablet_manager.cc:1905] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:32.694022 1658 tablet_service.cc:1468] Processing CreateTablet for tablet 5f5ce7fc87164ae1ace91557275285f8 (DEFAULT_TABLE table=default.table_1e68626ebae94b208424f5f5a66dd06e [id=aa8a6a6f408d454abdc5948e21f84233]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.696017 1792 tablet_service.cc:1468] Processing CreateTablet for tablet 5f5ce7fc87164ae1ace91557275285f8 (DEFAULT_TABLE table=default.table_1e68626ebae94b208424f5f5a66dd06e [id=aa8a6a6f408d454abdc5948e21f84233]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.696291 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5ce7fc87164ae1ace91557275285f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.696419 2075 ts_tablet_manager.cc:1918] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 14:04:32.696488 2075 log.cc:1199] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/6a9e685f802d47c5b8d79c17c6ff8427
I20250626 14:04:32.696774 2075 ts_tablet_manager.cc:1939] T 6a9e685f802d47c5b8d79c17c6ff8427 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:32.692512 1479 master_service.cc:948] ReplaceTablet: received request to replace tablet 6a9e685f802d47c5b8d79c17c6ff8427 from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:32.698462 1472 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet 6a9e685f802d47c5b8d79c17c6ff8427 (table default.table_1e68626ebae94b208424f5f5a66dd06e [id=aa8a6a6f408d454abdc5948e21f84233]) successfully deleted
I20250626 14:04:32.699817 1479 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_8a07386d09424c52995940144f8d5d3c" } modify_external_catalogs: true
I20250626 14:04:32.700029 1479 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_8a07386d09424c52995940144f8d5d3c" } modify_external_catalogs: true
I20250626 14:04:32.692709 2059 tablet_bootstrap.cc:492] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:32.701340 2059 tablet_bootstrap.cc:654] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.698666 1928 tablet_service.cc:1468] Processing CreateTablet for tablet 5f5ce7fc87164ae1ace91557275285f8 (DEFAULT_TABLE table=default.table_1e68626ebae94b208424f5f5a66dd06e [id=aa8a6a6f408d454abdc5948e21f84233]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.702184 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5ce7fc87164ae1ace91557275285f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.702208 1477 catalog_manager.cc:2149] created HMS catalog entry for table [id=61be84e63b2e478b9da5619fdbdffee9]
W20250626 14:04:32.704849 1483 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3a3f21b3c0e54e438561e02ab79fa36b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 14:04:32.705621 1486 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_22f5000aea364833a97948c4478f7e27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 14:04:32.706272 1482 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e6e7d4c291af4d29baf43e204c5292f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 14:04:32.706928 1481 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fb8ebc7aa8894f28b4d79252382f356a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 14:04:32.707041 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5ce7fc87164ae1ace91557275285f8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 14:04:32.707563 1480 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0be8dc44c93040319885961d6e4a0608 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 14:04:32.708218 1478 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f9b5b5e2dcf64f9e889c618242274d25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 14:04:32.709149 1477 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_72f9307d2f5247a0ae4499c953ea1650" } modify_external_catalogs: true
I20250626 14:04:32.709267 1477 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_72f9307d2f5247a0ae4499c953ea1650" } modify_external_catalogs: true
I20250626 14:04:32.710098 2061 tablet_bootstrap.cc:492] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:32.710301 2065 raft_consensus.cc:491] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:32.710455 2065 raft_consensus.cc:513] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.711196 2061 tablet_bootstrap.cc:654] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.711756 2065 leader_election.cc:290] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:32.716141 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.253:44447
--webserver_interface=127.1.11.253
--webserver_port=37937
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250626 14:04:32.718144 2073 tablet_replica.cc:331] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:32.718683 2073 raft_consensus.cc:2241] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:32.719110 2073 raft_consensus.cc:2270] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:32.720389 2074 ts_tablet_manager.cc:1905] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:32.720388 2073 ts_tablet_manager.cc:1905] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:32.722239 1792 tablet_service.cc:1468] Processing CreateTablet for tablet 680301fb5722406194f4dcadeb726964 (DEFAULT_TABLE table=default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.722301 1658 tablet_service.cc:1468] Processing CreateTablet for tablet 680301fb5722406194f4dcadeb726964 (DEFAULT_TABLE table=default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.722558 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 680301fb5722406194f4dcadeb726964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.722568 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 680301fb5722406194f4dcadeb726964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.722937 2074 ts_tablet_manager.cc:1918] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 14:04:32.723017 2074 log.cc:1199] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/6a9e685f802d47c5b8d79c17c6ff8427
I20250626 14:04:32.723246 2074 ts_tablet_manager.cc:1939] T 6a9e685f802d47c5b8d79c17c6ff8427 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:32.723766 2068 raft_consensus.cc:491] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:32.723902 2068 raft_consensus.cc:513] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.724468 1472 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet 6a9e685f802d47c5b8d79c17c6ff8427 (table default.table_1e68626ebae94b208424f5f5a66dd06e [id=aa8a6a6f408d454abdc5948e21f84233]) successfully deleted
I20250626 14:04:32.725231 2068 leader_election.cc:290] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:32.727833 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703110ded5c348cb96e47b437f340375" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:32.728039 1677 raft_consensus.cc:2466] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:32.728756 1884 leader_election.cc:304] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:32.729130 2065 raft_consensus.cc:2802] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:32.729212 2065 raft_consensus.cc:491] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:32.729257 2065 raft_consensus.cc:3058] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:32.730386 2065 raft_consensus.cc:513] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.730804 2065 leader_election.cc:290] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Requested vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:32.731705 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703110ded5c348cb96e47b437f340375" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:32.731833 1677 raft_consensus.cc:3058] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:32.733037 1677 raft_consensus.cc:2466] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:32.733273 2061 tablet_bootstrap.cc:492] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:32.733367 2061 ts_tablet_manager.cc:1397] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.000s
I20250626 14:04:32.733377 1884 leader_election.cc:304] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:32.733577 2065 raft_consensus.cc:2802] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:32.733781 2065 raft_consensus.cc:695] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Running, Role: LEADER
I20250626 14:04:32.733863 2061 raft_consensus.cc:357] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.733964 2061 raft_consensus.cc:383] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.734002 2061 raft_consensus.cc:738] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.734048 2065 consensus_queue.cc:237] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.734122 2061 consensus_queue.cc:260] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.734388 2061 ts_tablet_manager.cc:1428] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:32.734576 2061 tablet_bootstrap.cc:492] T 680301fb5722406194f4dcadeb726964 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:32.735633 2061 tablet_bootstrap.cc:654] T 680301fb5722406194f4dcadeb726964 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.737603 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703110ded5c348cb96e47b437f340375" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:32.728477 1928 tablet_service.cc:1468] Processing CreateTablet for tablet 680301fb5722406194f4dcadeb726964 (DEFAULT_TABLE table=default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.737890 1813 raft_consensus.cc:2466] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:32.738113 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 680301fb5722406194f4dcadeb726964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.738168 1812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703110ded5c348cb96e47b437f340375" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:32.738255 1812 raft_consensus.cc:3058] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:32.739636 1812 raft_consensus.cc:2466] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:32.741796 2059 tablet_bootstrap.cc:492] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:32.741899 2059 ts_tablet_manager.cc:1397] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.000s
I20250626 14:04:32.742425 2059 raft_consensus.cc:357] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.742570 2059 raft_consensus.cc:383] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.742610 2059 raft_consensus.cc:738] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.742736 2059 consensus_queue.cc:260] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.743126 2059 ts_tablet_manager.cc:1428] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 14:04:32.743359 2059 tablet_bootstrap.cc:492] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:32.745229 2060 tablet_bootstrap.cc:492] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:32.745319 2060 ts_tablet_manager.cc:1397] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.001s
I20250626 14:04:32.745833 2060 raft_consensus.cc:357] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.745951 2060 raft_consensus.cc:383] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.745990 2060 raft_consensus.cc:738] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.746138 2073 ts_tablet_manager.cc:1918] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 14:04:32.746129 2060 consensus_queue.cc:260] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.746224 2073 log.cc:1199] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/6a9e685f802d47c5b8d79c17c6ff8427
I20250626 14:04:32.746410 2060 ts_tablet_manager.cc:1428] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 14:04:32.746647 2060 tablet_bootstrap.cc:492] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:32.746848 2073 ts_tablet_manager.cc:1939] T 6a9e685f802d47c5b8d79c17c6ff8427 P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:32.747674 2060 tablet_bootstrap.cc:654] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.748212 1473 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet 6a9e685f802d47c5b8d79c17c6ff8427 (table default.table_1e68626ebae94b208424f5f5a66dd06e [id=aa8a6a6f408d454abdc5948e21f84233]) successfully deleted
I20250626 14:04:32.749290 1812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34cd6172930d497698cb9239220566af" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:32.750499 1812 raft_consensus.cc:2466] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 0.
I20250626 14:04:32.744520 2059 tablet_bootstrap.cc:654] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.751713 1613 leader_election.cc:304] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:32.752000 2068 raft_consensus.cc:2802] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:32.752076 2068 raft_consensus.cc:491] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:32.752115 2068 raft_consensus.cc:3058] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:32.753271 2068 raft_consensus.cc:513] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.753655 2068 leader_election.cc:290] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 election: Requested vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:32.754840 1948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34cd6172930d497698cb9239220566af" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:32.755074 1948 raft_consensus.cc:2466] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 0.
I20250626 14:04:32.755379 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34cd6172930d497698cb9239220566af" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:32.755491 1947 raft_consensus.cc:3058] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:32.756633 1947 raft_consensus.cc:2466] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
I20250626 14:04:32.757356 1812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34cd6172930d497698cb9239220566af" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:32.757500 1812 raft_consensus.cc:3058] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:32.758612 1812 raft_consensus.cc:2466] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
W20250626 14:04:32.758913 1997 tablet.cc:2378] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 14:04:32.759845 1613 leader_election.cc:304] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:32.760047 2068 raft_consensus.cc:2802] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:32.760141 2068 raft_consensus.cc:695] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [term 1 LEADER]: Becoming Leader. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Running, Role: LEADER
I20250626 14:04:32.760411 2068 consensus_queue.cc:237] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.768042 2061 tablet_bootstrap.cc:492] T 680301fb5722406194f4dcadeb726964 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:32.768151 2061 ts_tablet_manager.cc:1397] T 680301fb5722406194f4dcadeb726964 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.001s
I20250626 14:04:32.769845 2061 raft_consensus.cc:357] T 680301fb5722406194f4dcadeb726964 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.770037 2061 raft_consensus.cc:383] T 680301fb5722406194f4dcadeb726964 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.770077 2061 raft_consensus.cc:738] T 680301fb5722406194f4dcadeb726964 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.770200 2061 consensus_queue.cc:260] T 680301fb5722406194f4dcadeb726964 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.770467 2061 ts_tablet_manager.cc:1428] T 680301fb5722406194f4dcadeb726964 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 14:04:32.773877 2060 tablet_bootstrap.cc:492] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:32.773995 2060 ts_tablet_manager.cc:1397] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.001s
I20250626 14:04:32.773963 2050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:32.774092 2050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:32.774503 2060 raft_consensus.cc:357] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.774623 2060 raft_consensus.cc:383] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.774659 2060 raft_consensus.cc:738] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.774787 2060 consensus_queue.cc:260] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.775061 2060 ts_tablet_manager.cc:1428] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 14:04:32.775283 2060 tablet_bootstrap.cc:492] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:32.776392 2060 tablet_bootstrap.cc:654] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.779604 2059 tablet_bootstrap.cc:492] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:32.779731 2059 ts_tablet_manager.cc:1397] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20250626 14:04:32.780274 2059 raft_consensus.cc:357] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.780464 2059 raft_consensus.cc:383] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.780507 2059 raft_consensus.cc:738] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.780681 2059 consensus_queue.cc:260] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.780982 2059 ts_tablet_manager.cc:1428] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:32.781181 2059 tablet_bootstrap.cc:492] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:32.782373 2060 tablet_bootstrap.cc:492] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:32.782481 2060 ts_tablet_manager.cc:1397] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20250626 14:04:32.782559 2059 tablet_bootstrap.cc:654] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.783007 2060 raft_consensus.cc:357] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.783131 2060 raft_consensus.cc:383] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.783174 2060 raft_consensus.cc:738] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.783310 2060 consensus_queue.cc:260] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.783573 2060 ts_tablet_manager.cc:1428] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:32.784281 1483 catalog_manager.cc:2149] created HMS catalog entry for table [id=fb8ec39fc8a34deca5b53c18cc207140]
I20250626 14:04:32.790782 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_0b500f44e16e45e589c01ab7ad9a621b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:32.791769 2059 tablet_bootstrap.cc:492] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:32.791890 2059 ts_tablet_manager.cc:1397] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250626 14:04:32.794420 2059 raft_consensus.cc:357] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.794579 2059 raft_consensus.cc:383] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.794620 2059 raft_consensus.cc:738] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.797123 1928 tablet_service.cc:1468] Processing CreateTablet for tablet f81ab56968784be1b14a892cffc7b53b (DEFAULT_TABLE table=default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.797123 1792 tablet_service.cc:1468] Processing CreateTablet for tablet f81ab56968784be1b14a892cffc7b53b (DEFAULT_TABLE table=default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.797430 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f81ab56968784be1b14a892cffc7b53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.797482 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f81ab56968784be1b14a892cffc7b53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.799693 2060 tablet_bootstrap.cc:492] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:32.800761 1658 tablet_service.cc:1468] Processing CreateTablet for tablet f81ab56968784be1b14a892cffc7b53b (DEFAULT_TABLE table=default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.801098 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f81ab56968784be1b14a892cffc7b53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.801098 2060 tablet_bootstrap.cc:654] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.803068 2061 tablet_bootstrap.cc:492] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:32.794826 2059 consensus_queue.cc:260] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.804234 2061 tablet_bootstrap.cc:654] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.804968 2083 raft_consensus.cc:491] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:32.805087 2083 raft_consensus.cc:513] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.805553 2083 leader_election.cc:290] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:32.805925 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f5ce7fc87164ae1ace91557275285f8" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:32.806074 1677 raft_consensus.cc:2466] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:32.806393 1884 leader_election.cc:304] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:32.807164 2069 raft_consensus.cc:491] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:32.807271 2069 raft_consensus.cc:513] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.807287 1812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f5ce7fc87164ae1ace91557275285f8" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:32.807471 2083 raft_consensus.cc:2802] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:32.807528 2083 raft_consensus.cc:491] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:32.807559 2083 raft_consensus.cc:3058] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:32.807641 1812 raft_consensus.cc:2466] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:32.808403 2069 leader_election.cc:290] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:32.808681 2083 raft_consensus.cc:513] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.809093 2083 leader_election.cc:290] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Requested vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:32.812278 1812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f5ce7fc87164ae1ace91557275285f8" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:32.812477 1812 raft_consensus.cc:3058] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:32.813977 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f5ce7fc87164ae1ace91557275285f8" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:32.814154 1677 raft_consensus.cc:3058] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:32.817286 1677 raft_consensus.cc:2466] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:32.818147 1884 leader_election.cc:304] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:32.818974 1812 raft_consensus.cc:2466] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:32.822041 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f5ce7fc87164ae1ace91557275285f8" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:32.822265 1947 raft_consensus.cc:2391] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e517893e13f546c0b40d976633b49651 in current term 1: Already voted for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in this term.
I20250626 14:04:32.823110 2083 raft_consensus.cc:2802] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:32.823220 2083 raft_consensus.cc:695] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Running, Role: LEADER
I20250626 14:04:32.823374 2083 consensus_queue.cc:237] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
W20250626 14:04:32.826471 1862 tablet.cc:2378] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 14:04:32.827486 2059 ts_tablet_manager.cc:1428] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20250626 14:04:32.829846 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f5ce7fc87164ae1ace91557275285f8" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:32.830076 1677 raft_consensus.cc:2391] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e517893e13f546c0b40d976633b49651 in current term 1: Already voted for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in this term.
I20250626 14:04:32.830538 2059 tablet_bootstrap.cc:492] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:32.830559 1749 leader_election.cc:304] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e517893e13f546c0b40d976633b49651; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b
I20250626 14:04:32.831000 2069 raft_consensus.cc:2747] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 14:04:32.831843 2059 tablet_bootstrap.cc:654] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.833034 2059 tablet_bootstrap.cc:492] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:32.833137 2059 ts_tablet_manager.cc:1397] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:32.833590 2059 raft_consensus.cc:357] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.833707 2059 raft_consensus.cc:383] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.833755 2059 raft_consensus.cc:738] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.833917 2059 consensus_queue.cc:260] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.834477 2059 ts_tablet_manager.cc:1428] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:32.835933 2060 tablet_bootstrap.cc:492] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:32.836043 2060 ts_tablet_manager.cc:1397] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20250626 14:04:32.836100 2061 tablet_bootstrap.cc:492] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:32.836194 2061 ts_tablet_manager.cc:1397] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20250626 14:04:32.836608 2060 raft_consensus.cc:357] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.836727 2060 raft_consensus.cc:383] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.836779 2060 raft_consensus.cc:738] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.836730 2061 raft_consensus.cc:357] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.836839 2061 raft_consensus.cc:383] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.836877 2061 raft_consensus.cc:738] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.836925 2060 consensus_queue.cc:260] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.836993 2061 consensus_queue.cc:260] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.837210 2060 ts_tablet_manager.cc:1428] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:32.837265 2061 ts_tablet_manager.cc:1428] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:32.844615 2069 raft_consensus.cc:491] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:32.844750 2069 raft_consensus.cc:513] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.845201 2069 leader_election.cc:290] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:32.845616 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "680301fb5722406194f4dcadeb726964" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:32.845657 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "680301fb5722406194f4dcadeb726964" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:32.845777 1947 raft_consensus.cc:2466] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:32.845777 1677 raft_consensus.cc:2466] T 680301fb5722406194f4dcadeb726964 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:32.846120 1748 leader_election.cc:304] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:32.846343 2069 raft_consensus.cc:2802] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:32.846410 2069 raft_consensus.cc:491] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:32.846467 2069 raft_consensus.cc:3058] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:32.847340 2069 raft_consensus.cc:513] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.847769 2069 leader_election.cc:290] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Requested vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:32.848150 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "680301fb5722406194f4dcadeb726964" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:32.848269 1677 raft_consensus.cc:3058] T 680301fb5722406194f4dcadeb726964 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:32.849292 1677 raft_consensus.cc:2466] T 680301fb5722406194f4dcadeb726964 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:32.849721 1749 leader_election.cc:304] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:32.850277 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "680301fb5722406194f4dcadeb726964" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:32.850419 1947 raft_consensus.cc:3058] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:32.851388 1947 raft_consensus.cc:2466] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:32.851929 2069 raft_consensus.cc:2802] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:32.852160 2069 raft_consensus.cc:695] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Becoming Leader. State: Replica: e517893e13f546c0b40d976633b49651, State: Running, Role: LEADER
I20250626 14:04:32.852438 2069 consensus_queue.cc:237] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.876787 1486 catalog_manager.cc:2149] created HMS catalog entry for table [id=8bc8b25d9539413ea196c158d55abe80]
I20250626 14:04:32.882956 1486 catalog_manager.cc:5582] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193). New cstate: current_term: 1 leader_uuid: "4e57adb83e9f451488ed1d0034043e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250626 14:04:32.899118 1658 tablet_service.cc:1468] Processing CreateTablet for tablet f51154f6fd584a95976d6af90f9133e9 (DEFAULT_TABLE table=default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.899519 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51154f6fd584a95976d6af90f9133e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.900169 1486 catalog_manager.cc:5582] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195). New cstate: current_term: 1 leader_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250626 14:04:32.900951 1928 tablet_service.cc:1468] Processing CreateTablet for tablet f51154f6fd584a95976d6af90f9133e9 (DEFAULT_TABLE table=default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.901237 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51154f6fd584a95976d6af90f9133e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.903652 2059 tablet_bootstrap.cc:492] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:32.905046 1792 tablet_service.cc:1468] Processing CreateTablet for tablet f51154f6fd584a95976d6af90f9133e9 (DEFAULT_TABLE table=default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.905401 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51154f6fd584a95976d6af90f9133e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.906471 2059 tablet_bootstrap.cc:654] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.908176 2060 tablet_bootstrap.cc:492] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:32.909523 2060 tablet_bootstrap.cc:654] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.909770 2061 tablet_bootstrap.cc:492] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:32.910997 2061 tablet_bootstrap.cc:654] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.911401 2060 tablet_bootstrap.cc:492] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:32.911504 2060 ts_tablet_manager.cc:1397] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:32.912087 2060 raft_consensus.cc:357] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.912227 2060 raft_consensus.cc:383] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.912271 2060 raft_consensus.cc:738] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.912467 2060 consensus_queue.cc:260] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.913599 2060 ts_tablet_manager.cc:1428] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:32.914456 2059 tablet_bootstrap.cc:492] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:32.914561 2059 ts_tablet_manager.cc:1397] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250626 14:04:32.915105 2059 raft_consensus.cc:357] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.915211 2059 raft_consensus.cc:383] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.915246 2059 raft_consensus.cc:738] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.915930 2059 consensus_queue.cc:260] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.918550 2059 ts_tablet_manager.cc:1428] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250626 14:04:32.920579 1482 catalog_manager.cc:2149] created HMS catalog entry for table [id=c3aa594eee4244029a38801f34b07447]
I20250626 14:04:32.922420 2061 tablet_bootstrap.cc:492] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:32.922540 2061 ts_tablet_manager.cc:1397] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250626 14:04:32.923144 2061 raft_consensus.cc:357] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.923262 2061 raft_consensus.cc:383] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.923309 2061 raft_consensus.cc:738] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.925261 2061 consensus_queue.cc:260] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.931502 1482 catalog_manager.cc:5582] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195). New cstate: current_term: 1 leader_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250626 14:04:32.932744 2061 ts_tablet_manager.cc:1428] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
W20250626 14:04:32.937191 2080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:32.937759 2080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:32.937830 2080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250626 14:04:32.938369 1482 catalog_manager.cc:5582] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 reported cstate change: term changed from 0 to 1, leader changed from <none> to e517893e13f546c0b40d976633b49651 (127.1.11.194). New cstate: current_term: 1 leader_uuid: "e517893e13f546c0b40d976633b49651" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } health_report { overall_health: UNKNOWN } } }
W20250626 14:04:32.942596 2080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:04:32.942682 2080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:32.942710 2080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:04:32.942735 2080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:04:32.943867 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_3a3f21b3c0e54e438561e02ab79fa36b" } new_table_name: "default.Table_3633fc38c5374180a081fab63a1eed84" modify_external_catalogs: true
I20250626 14:04:32.948266 2080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.253:44447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.11.253
--webserver_port=37937
--never_fsync=true
--heap_profile_path=/tmp/kudu.2080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:32.949771 2080 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:32.950126 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_33ee3521f64d4cefb8856259510e5f0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:32.951295 2080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 14:04:32.951407 1792 tablet_service.cc:1468] Processing CreateTablet for tablet f7fa3619f0c541bf8e83db2d4fd24ced (DEFAULT_TABLE table=default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.951808 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7fa3619f0c541bf8e83db2d4fd24ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.954322 2060 tablet_bootstrap.cc:492] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:32.954584 1928 tablet_service.cc:1468] Processing CreateTablet for tablet f7fa3619f0c541bf8e83db2d4fd24ced (DEFAULT_TABLE table=default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.954891 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7fa3619f0c541bf8e83db2d4fd24ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.955473 2060 tablet_bootstrap.cc:654] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.957316 2059 tablet_bootstrap.cc:492] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:32.958487 2059 tablet_bootstrap.cc:654] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.958778 1658 tablet_service.cc:1468] Processing CreateTablet for tablet f7fa3619f0c541bf8e83db2d4fd24ced (DEFAULT_TABLE table=default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.959081 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7fa3619f0c541bf8e83db2d4fd24ced. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 14:04:32.961247 2110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:32.961618 2059 tablet_bootstrap.cc:492] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
W20250626 14:04:32.961647 2112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:32.961714 2059 ts_tablet_manager.cc:1397] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250626 14:04:32.962328 2059 raft_consensus.cc:357] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.962472 2059 raft_consensus.cc:383] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.962512 2059 raft_consensus.cc:738] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.962657 2059 consensus_queue.cc:260] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.962833 2061 tablet_bootstrap.cc:492] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:32.962987 2059 ts_tablet_manager.cc:1428] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:32.964234 2061 tablet_bootstrap.cc:654] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.964356 2080 server_base.cc:1048] running on GCE node
W20250626 14:04:32.965391 2109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:32.966162 2080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:32.967461 2080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:32.968279 2061 tablet_bootstrap.cc:492] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:32.968283 2060 tablet_bootstrap.cc:492] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:32.968392 2061 ts_tablet_manager.cc:1397] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250626 14:04:32.968392 2060 ts_tablet_manager.cc:1397] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20250626 14:04:32.968961 2061 raft_consensus.cc:357] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.968961 2060 raft_consensus.cc:357] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.969089 2061 raft_consensus.cc:383] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.969089 2060 raft_consensus.cc:383] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.969126 2061 raft_consensus.cc:738] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.969126 2060 raft_consensus.cc:738] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.969265 2060 consensus_queue.cc:260] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.969265 2061 consensus_queue.cc:260] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.969554 2060 ts_tablet_manager.cc:1428] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:32.969571 2061 ts_tablet_manager.cc:1428] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:32.969821 2080 hybrid_clock.cc:648] HybridClock initialized: now 1750946672969749 us; error 87 us; skew 500 ppm
I20250626 14:04:32.973773 2080 webserver.cc:469] Webserver started at http://127.1.11.253:37937/ using document root <none> and password file <none>
I20250626 14:04:32.974057 1481 catalog_manager.cc:2149] created HMS catalog entry for table [id=9177a82a73394ff798d7735ed15f9e75]
I20250626 14:04:32.974611 2080 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:32.974705 2080 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:32.981683 1481 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_566f55c5d8c34043a37041fef8cb9f45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:32.984308 2080 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20250626 14:04:32.987061 2120 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:32.988327 2080 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:32.988808 1792 tablet_service.cc:1468] Processing CreateTablet for tablet d2a95d2f060943a68721021a204e0cc3 (DEFAULT_TABLE table=default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.988464 2080 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
uuid: "07532b7648e4414789ed2b099bad2b19"
format_stamp: "Formatted at 2025-06-26 14:04:30 on dist-test-slave-rptj"
I20250626 14:04:32.988976 2080 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:04:32.989143 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a95d2f060943a68721021a204e0cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.990798 1928 tablet_service.cc:1468] Processing CreateTablet for tablet d2a95d2f060943a68721021a204e0cc3 (DEFAULT_TABLE table=default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.991081 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a95d2f060943a68721021a204e0cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.991618 2060 tablet_bootstrap.cc:492] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:32.992843 2060 tablet_bootstrap.cc:654] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.993119 1658 tablet_service.cc:1468] Processing CreateTablet for tablet d2a95d2f060943a68721021a204e0cc3 (DEFAULT_TABLE table=default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:32.993364 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a95d2f060943a68721021a204e0cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:32.993391 2059 tablet_bootstrap.cc:492] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:32.994365 2059 tablet_bootstrap.cc:654] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.995551 2061 tablet_bootstrap.cc:492] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:32.996616 2061 tablet_bootstrap.cc:654] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:32.997855 2061 tablet_bootstrap.cc:492] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:32.997957 2061 ts_tablet_manager.cc:1397] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250626 14:04:32.998577 2061 raft_consensus.cc:357] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.998715 2061 raft_consensus.cc:383] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:32.998761 2061 raft_consensus.cc:738] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:32.998951 2061 consensus_queue.cc:260] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:32.999977 2059 tablet_bootstrap.cc:492] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:33.000065 2059 ts_tablet_manager.cc:1397] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20250626 14:04:33.000043 2060 tablet_bootstrap.cc:492] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:33.000185 2060 ts_tablet_manager.cc:1397] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250626 14:04:33.000559 2059 raft_consensus.cc:357] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.000658 2059 raft_consensus.cc:383] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.000686 2059 raft_consensus.cc:738] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.000818 2059 consensus_queue.cc:260] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.000785 2060 raft_consensus.cc:357] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.000903 2060 raft_consensus.cc:383] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.000943 2060 raft_consensus.cc:738] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.001075 2060 consensus_queue.cc:260] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.001368 2060 ts_tablet_manager.cc:1428] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:33.003227 2061 ts_tablet_manager.cc:1428] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250626 14:04:33.003782 2059 ts_tablet_manager.cc:1428] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250626 14:04:33.021237 2080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:33.022112 2080 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:33.022318 2080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:33.022984 1480 catalog_manager.cc:2149] created HMS catalog entry for table [id=2339533510b347b9bed82553187bab03]
I20250626 14:04:33.029217 1480 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_22f5000aea364833a97948c4478f7e27" } new_table_name: "default.Table_8268b369fb20436fb4e562f6a97eee1d" modify_external_catalogs: true
I20250626 14:04:33.033757 2080 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.253:44447
I20250626 14:04:33.035357 2080 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
I20250626 14:04:33.035422 2175 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.253:44447 every 8 connection(s)
I20250626 14:04:33.038224 1658 tablet_service.cc:1468] Processing CreateTablet for tablet dd5f0eb6de154b3c9ffdd6d8f7042a49 (DEFAULT_TABLE table=default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.038537 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd5f0eb6de154b3c9ffdd6d8f7042a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.038543 1792 tablet_service.cc:1468] Processing CreateTablet for tablet dd5f0eb6de154b3c9ffdd6d8f7042a49 (DEFAULT_TABLE table=default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.038816 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd5f0eb6de154b3c9ffdd6d8f7042a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.040211 2176 sys_catalog.cc:263] Verifying existing consensus state
I20250626 14:04:33.040794 2061 tablet_bootstrap.cc:492] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:33.041020 1928 tablet_service.cc:1468] Processing CreateTablet for tablet dd5f0eb6de154b3c9ffdd6d8f7042a49 (DEFAULT_TABLE table=default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.041267 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd5f0eb6de154b3c9ffdd6d8f7042a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.041797 2061 tablet_bootstrap.cc:654] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.042176 2176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Bootstrap starting.
I20250626 14:04:33.042783 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 2080
I20250626 14:04:33.043536 2060 tablet_bootstrap.cc:492] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:33.044708 2060 tablet_bootstrap.cc:654] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.045390 2059 tablet_bootstrap.cc:492] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:33.046260 2059 tablet_bootstrap.cc:654] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.047874 2059 tablet_bootstrap.cc:492] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:33.047955 2059 ts_tablet_manager.cc:1397] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:33.048403 2059 raft_consensus.cc:357] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.048520 2059 raft_consensus.cc:383] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.048599 2059 raft_consensus.cc:738] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.048722 2059 consensus_queue.cc:260] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.048995 2059 ts_tablet_manager.cc:1428] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:33.050491 2176 log.cc:826] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Log is configured to *not* fsync() on all Append() calls
I20250626 14:04:33.051841 2060 tablet_bootstrap.cc:492] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:33.051942 2060 ts_tablet_manager.cc:1397] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250626 14:04:33.052474 2060 raft_consensus.cc:357] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.052609 2060 raft_consensus.cc:383] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.052647 2060 raft_consensus.cc:738] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.052793 2060 consensus_queue.cc:260] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.053090 2060 ts_tablet_manager.cc:1428] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:33.054050 2061 tablet_bootstrap.cc:492] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:33.054172 2061 ts_tablet_manager.cc:1397] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20250626 14:04:33.054819 2061 raft_consensus.cc:357] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.054927 2061 raft_consensus.cc:383] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.054960 2061 raft_consensus.cc:738] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.055079 2061 consensus_queue.cc:260] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.055351 2061 ts_tablet_manager.cc:1428] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:33.063094 2176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 14:04:33.063618 2176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Bootstrap complete.
I20250626 14:04:33.067216 2176 raft_consensus.cc:357] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:33.067791 2176 raft_consensus.cc:738] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07532b7648e4414789ed2b099bad2b19, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.067826 1996 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:33.068400 2176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:33.069770 2176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:04:33.070552 1726 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:33.075557 1861 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:33.075656 1996 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:33.077355 2140 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e517893e13f546c0b40d976633b49651" instance_seqno: 1750946671821196) as {username='slave'} at 127.1.11.194:35821; Asking this server to re-register.
I20250626 14:04:33.077984 1861 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:33.078636 2138 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" instance_seqno: 1750946672002466) as {username='slave'} at 127.1.11.195:35515; Asking this server to re-register.
I20250626 14:04:33.078969 2200 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:33.079006 2200 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: loading cluster ID for follower catalog manager: success
I20250626 14:04:33.079106 1996 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:33.079380 2137 ts_manager.cc:194] Registered new tserver with Master: e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:33.080118 2138 ts_manager.cc:194] Registered new tserver with Master: 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:33.081362 2200 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: acquiring CA information for follower catalog manager: success
I20250626 14:04:33.082876 2200 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 14:04:33.083470 2186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:33.083585 2186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:33.088336 1478 catalog_manager.cc:2149] created HMS catalog entry for table [id=457542e389754e82bfcab4452db8557a]
I20250626 14:04:33.092693 1478 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51198 (request call id 9) took 1042 ms. Trace:
I20250626 14:04:33.092810 1478 rpcz_store.cc:276] 0626 14:04:32.049991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 14:04:32.051719 (+ 1728us) service_pool.cc:225] Handling call
0626 14:04:32.708288 (+656569us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 14:04:32.708585 (+ 297us) catalog_manager.cc:2122] Created new table and tablet info
0626 14:04:33.088326 (+379741us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 14:04:33.092464 (+ 4138us) catalog_manager.cc:2182] Wrote table and tablets to system table
0626 14:04:33.092588 (+ 124us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0626 14:04:33.092672 (+ 84us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 14:04:33.088930 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 14:04:33.089044 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0626 14:04:33.089049 (+ 5us) write_op.cc:435] Acquired schema lock
0626 14:04:33.089051 (+ 2us) tablet.cc:660] Decoding operations
0626 14:04:33.089123 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0626 14:04:33.089134 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0626 14:04:33.089137 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0626 14:04:33.089193 (+ 56us) tablet.cc:699] Row locks acquired
0626 14:04:33.089195 (+ 2us) write_op.cc:260] PREPARE: finished
0626 14:04:33.089226 (+ 31us) write_op.cc:270] Start()
0626 14:04:33.089255 (+ 29us) write_op.cc:276] Timestamp: P: 1750946673089169 usec, L: 0
0626 14:04:33.089257 (+ 2us) op_driver.cc:348] REPLICATION: starting
0626 14:04:33.089412 (+ 155us) log.cc:844] Serialized 432 byte log entry
0626 14:04:33.091873 (+ 2461us) op_driver.cc:464] REPLICATION: finished
0626 14:04:33.091991 (+ 118us) write_op.cc:301] APPLY: starting
0626 14:04:33.092006 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0626 14:04:33.092037 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0626 14:04:33.092051 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 14:04:33.092124 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 14:04:33.092153 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0626 14:04:33.092157 (+ 4us) write_op.cc:312] APPLY: finished
0626 14:04:33.092214 (+ 57us) log.cc:844] Serialized 28 byte log entry
0626 14:04:33.092264 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0626 14:04:33.092277 (+ 13us) write_op.cc:454] Released schema lock
0626 14:04:33.092305 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0626 14:04:33.092370 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0626 14:04:33.092373 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":314304,"Hive Metastore.run_cpu_time_us":232,"Hive Metastore.run_wall_time_us":65316,"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":369,"apply.run_wall_time_us":368,"num_ops":2,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":585,"prepare.run_wall_time_us":585,"raft.queue_time_us":694,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":174,"replication_time_us":2588}]]}
I20250626 14:04:33.099861 1928 tablet_service.cc:1468] Processing CreateTablet for tablet 071dcb923e314fca9fbbf94592b01fe2 (DEFAULT_TABLE table=default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.100088 1792 tablet_service.cc:1468] Processing CreateTablet for tablet 071dcb923e314fca9fbbf94592b01fe2 (DEFAULT_TABLE table=default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.100210 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 071dcb923e314fca9fbbf94592b01fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.100390 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 071dcb923e314fca9fbbf94592b01fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.101413 1658 tablet_service.cc:1468] Processing CreateTablet for tablet 071dcb923e314fca9fbbf94592b01fe2 (DEFAULT_TABLE table=default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.101679 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 071dcb923e314fca9fbbf94592b01fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.102645 2060 tablet_bootstrap.cc:492] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:33.102646 2059 tablet_bootstrap.cc:492] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:33.103674 2060 tablet_bootstrap.cc:654] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.103715 2059 tablet_bootstrap.cc:654] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.103902 2061 tablet_bootstrap.cc:492] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:33.104923 2061 tablet_bootstrap.cc:654] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.105253 2099 raft_consensus.cc:491] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:33.105333 2099 raft_consensus.cc:513] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.105775 2099 leader_election.cc:290] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:33.106149 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a95d2f060943a68721021a204e0cc3" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:33.106288 1677 raft_consensus.cc:2466] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:33.106596 2061 tablet_bootstrap.cc:492] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:33.106626 1749 leader_election.cc:304] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:33.106683 2061 ts_tablet_manager.cc:1397] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:33.106947 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a95d2f060943a68721021a204e0cc3" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:33.107087 1947 raft_consensus.cc:2466] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:33.107182 2061 raft_consensus.cc:357] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.107286 2061 raft_consensus.cc:383] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.107327 2061 raft_consensus.cc:738] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.107489 2061 consensus_queue.cc:260] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.107602 2099 raft_consensus.cc:2802] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:33.107671 2099 raft_consensus.cc:491] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:33.107719 2099 raft_consensus.cc:3058] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.107766 2061 ts_tablet_manager.cc:1428] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 14:04:33.108791 2099 raft_consensus.cc:513] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.109150 2099 leader_election.cc:290] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Requested vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:33.109717 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a95d2f060943a68721021a204e0cc3" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:33.109834 1947 raft_consensus.cc:3058] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.110811 1947 raft_consensus.cc:2466] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:33.111044 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2a95d2f060943a68721021a204e0cc3" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:33.111160 1677 raft_consensus.cc:3058] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.111147 1478 master_service.cc:948] ReplaceTablet: received request to replace tablet f81ab56968784be1b14a892cffc7b53b from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:33.111871 1677 raft_consensus.cc:2466] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:33.112350 2138 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" instance_seqno: 1750946671637378) as {username='slave'} at 127.1.11.193:34829; Asking this server to re-register.
I20250626 14:04:33.112967 2060 tablet_bootstrap.cc:492] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:33.113065 2060 ts_tablet_manager.cc:1397] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250626 14:04:33.113587 2060 raft_consensus.cc:357] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.113687 2060 raft_consensus.cc:383] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.113718 2060 raft_consensus.cc:738] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.113839 2060 consensus_queue.cc:260] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.114063 1748 leader_election.cc:304] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:33.114116 2060 ts_tablet_manager.cc:1428] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:33.114890 2099 raft_consensus.cc:2802] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:33.115096 2099 raft_consensus.cc:695] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Becoming Leader. State: Replica: e517893e13f546c0b40d976633b49651, State: Running, Role: LEADER
I20250626 14:04:33.115216 2099 consensus_queue.cc:237] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.116371 1726 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:33.117551 2137 ts_manager.cc:194] Registered new tserver with Master: 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:33.120728 2059 tablet_bootstrap.cc:492] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:33.120849 2059 ts_tablet_manager.cc:1397] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20250626 14:04:33.121408 2059 raft_consensus.cc:357] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.121548 2059 raft_consensus.cc:383] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.121592 2059 raft_consensus.cc:738] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.121728 2059 consensus_queue.cc:260] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.122035 2059 ts_tablet_manager.cc:1428] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:33.122637 1478 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Sending DeleteTablet for 3 replicas of tablet f81ab56968784be1b14a892cffc7b53b
I20250626 14:04:33.122987 1478 catalog_manager.cc:6652] ReplaceTablet: tablet f81ab56968784be1b14a892cffc7b53b deleted and replaced by tablet ace98628290843938dc5db70e5a43a37
I20250626 14:04:33.124663 1658 tablet_service.cc:1515] Processing DeleteTablet for tablet f81ab56968784be1b14a892cffc7b53b with delete_type TABLET_DATA_DELETED (replaced by tablet ace98628290843938dc5db70e5a43a37) from {username='slave'} at 127.0.0.1:43706
I20250626 14:04:33.124950 2073 tablet_replica.cc:331] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:33.125092 2073 raft_consensus.cc:2241] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:33.125195 2073 raft_consensus.cc:2270] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:33.126205 2073 ts_tablet_manager.cc:1905] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:33.127046 1792 tablet_service.cc:1515] Processing DeleteTablet for tablet f81ab56968784be1b14a892cffc7b53b with delete_type TABLET_DATA_DELETED (replaced by tablet ace98628290843938dc5db70e5a43a37) from {username='slave'} at 127.0.0.1:48162
I20250626 14:04:33.127221 2075 tablet_replica.cc:331] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:33.127347 2075 raft_consensus.cc:2241] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:33.127468 2075 raft_consensus.cc:2270] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:33.128115 2075 ts_tablet_manager.cc:1905] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:33.128618 1928 tablet_service.cc:1515] Processing DeleteTablet for tablet f81ab56968784be1b14a892cffc7b53b with delete_type TABLET_DATA_DELETED (replaced by tablet ace98628290843938dc5db70e5a43a37) from {username='slave'} at 127.0.0.1:54640
I20250626 14:04:33.128796 2074 tablet_replica.cc:331] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:33.128909 2074 raft_consensus.cc:2241] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:33.129050 2074 raft_consensus.cc:2270] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:33.130203 2075 ts_tablet_manager.cc:1918] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 14:04:33.130280 2075 log.cc:1199] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/f81ab56968784be1b14a892cffc7b53b
I20250626 14:04:33.130481 2075 ts_tablet_manager.cc:1939] T f81ab56968784be1b14a892cffc7b53b P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:33.130594 2074 ts_tablet_manager.cc:1905] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:33.131137 1472 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet f81ab56968784be1b14a892cffc7b53b (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]) successfully deleted
I20250626 14:04:33.132808 2074 ts_tablet_manager.cc:1918] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 14:04:33.132886 2074 log.cc:1199] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/f81ab56968784be1b14a892cffc7b53b
I20250626 14:04:33.133296 2074 ts_tablet_manager.cc:1939] T f81ab56968784be1b14a892cffc7b53b P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:33.133600 1928 tablet_service.cc:1468] Processing CreateTablet for tablet ace98628290843938dc5db70e5a43a37 (DEFAULT_TABLE table=default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.133868 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ace98628290843938dc5db70e5a43a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.133967 1792 tablet_service.cc:1468] Processing CreateTablet for tablet ace98628290843938dc5db70e5a43a37 (DEFAULT_TABLE table=default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.134220 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ace98628290843938dc5db70e5a43a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.134450 1478 master_service.cc:948] ReplaceTablet: received request to replace tablet f81ab56968784be1b14a892cffc7b53b from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:33.134708 1478 master_service.cc:948] ReplaceTablet: received request to replace tablet 34cd6172930d497698cb9239220566af from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:33.135888 1658 tablet_service.cc:1468] Processing CreateTablet for tablet ace98628290843938dc5db70e5a43a37 (DEFAULT_TABLE table=default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.136163 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ace98628290843938dc5db70e5a43a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.138659 2059 tablet_bootstrap.cc:492] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:33.139837 2059 tablet_bootstrap.cc:654] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.140561 2073 ts_tablet_manager.cc:1918] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 14:04:33.140648 2073 log.cc:1199] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/f81ab56968784be1b14a892cffc7b53b
I20250626 14:04:33.140862 2073 ts_tablet_manager.cc:1939] T f81ab56968784be1b14a892cffc7b53b P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:33.140880 1478 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Sending DeleteTablet for 3 replicas of tablet 34cd6172930d497698cb9239220566af
I20250626 14:04:33.141719 2061 tablet_bootstrap.cc:492] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:33.142737 2061 tablet_bootstrap.cc:654] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.142846 1478 catalog_manager.cc:6652] ReplaceTablet: tablet 34cd6172930d497698cb9239220566af deleted and replaced by tablet d0f427caf3744354ae36c40f257ec89c
I20250626 14:04:33.144300 2061 tablet_bootstrap.cc:492] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:33.144393 2061 ts_tablet_manager.cc:1397] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:33.144943 1473 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet f81ab56968784be1b14a892cffc7b53b (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]) successfully deleted
I20250626 14:04:33.144933 2061 raft_consensus.cc:357] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.145038 2061 raft_consensus.cc:383] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.145071 2061 raft_consensus.cc:738] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.145192 2061 consensus_queue.cc:260] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.145466 2061 ts_tablet_manager.cc:1428] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:33.145550 1928 tablet_service.cc:1515] Processing DeleteTablet for tablet 34cd6172930d497698cb9239220566af with delete_type TABLET_DATA_DELETED (replaced by tablet d0f427caf3744354ae36c40f257ec89c) from {username='slave'} at 127.0.0.1:54640
I20250626 14:04:33.145756 1658 tablet_service.cc:1515] Processing DeleteTablet for tablet 34cd6172930d497698cb9239220566af with delete_type TABLET_DATA_DELETED (replaced by tablet d0f427caf3744354ae36c40f257ec89c) from {username='slave'} at 127.0.0.1:43706
I20250626 14:04:33.147289 1792 tablet_service.cc:1515] Processing DeleteTablet for tablet 34cd6172930d497698cb9239220566af with delete_type TABLET_DATA_DELETED (replaced by tablet d0f427caf3744354ae36c40f257ec89c) from {username='slave'} at 127.0.0.1:48162
I20250626 14:04:33.147630 2075 tablet_replica.cc:331] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:33.147779 2075 raft_consensus.cc:2241] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:33.147903 2075 raft_consensus.cc:2270] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:33.148571 2075 ts_tablet_manager.cc:1905] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:33.149677 1658 tablet_service.cc:1468] Processing CreateTablet for tablet d0f427caf3744354ae36c40f257ec89c (DEFAULT_TABLE table=default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.149765 1792 tablet_service.cc:1468] Processing CreateTablet for tablet d0f427caf3744354ae36c40f257ec89c (DEFAULT_TABLE table=default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.149932 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f427caf3744354ae36c40f257ec89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.149966 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f427caf3744354ae36c40f257ec89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.150789 2075 ts_tablet_manager.cc:1918] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 14:04:33.150853 2075 log.cc:1199] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/34cd6172930d497698cb9239220566af
I20250626 14:04:33.151057 2075 ts_tablet_manager.cc:1939] T 34cd6172930d497698cb9239220566af P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:33.152014 1472 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet 34cd6172930d497698cb9239220566af (table default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738]) successfully deleted
I20250626 14:04:33.152117 2061 tablet_bootstrap.cc:492] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:33.153134 1478 catalog_manager.cc:5582] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 reported cstate change: term changed from 0 to 1, leader changed from <none> to e517893e13f546c0b40d976633b49651 (127.1.11.194). New cstate: current_term: 1 leader_uuid: "e517893e13f546c0b40d976633b49651" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250626 14:04:33.153395 2061 tablet_bootstrap.cc:654] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.154632 2061 tablet_bootstrap.cc:492] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:33.154731 2061 ts_tablet_manager.cc:1397] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:33.155229 2061 raft_consensus.cc:357] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.155336 2061 raft_consensus.cc:383] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.155378 2061 raft_consensus.cc:738] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.155537 2061 consensus_queue.cc:260] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.155814 2061 ts_tablet_manager.cc:1428] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:33.155836 1928 tablet_service.cc:1468] Processing CreateTablet for tablet d0f427caf3744354ae36c40f257ec89c (DEFAULT_TABLE table=default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.156116 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f427caf3744354ae36c40f257ec89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.147781 2060 tablet_bootstrap.cc:492] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:33.157864 2060 tablet_bootstrap.cc:654] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.158975 2060 tablet_bootstrap.cc:492] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:33.159061 2060 ts_tablet_manager.cc:1397] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250626 14:04:33.159560 2060 raft_consensus.cc:357] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.159675 2060 raft_consensus.cc:383] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.159715 2060 raft_consensus.cc:738] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.159826 2060 consensus_queue.cc:260] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.159958 1472 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet f81ab56968784be1b14a892cffc7b53b (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]) successfully deleted
I20250626 14:04:33.160070 2060 ts_tablet_manager.cc:1428] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 14:04:33.160682 2060 tablet_bootstrap.cc:492] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:33.161687 2060 tablet_bootstrap.cc:654] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.162748 2060 tablet_bootstrap.cc:492] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:33.162823 2060 ts_tablet_manager.cc:1397] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:33.163298 2060 raft_consensus.cc:357] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.163444 2060 raft_consensus.cc:383] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.163478 2060 raft_consensus.cc:738] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.163612 2060 consensus_queue.cc:260] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.163903 2060 ts_tablet_manager.cc:1428] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 14:04:33.164052 1478 master_service.cc:948] ReplaceTablet: received request to replace tablet 703110ded5c348cb96e47b437f340375 from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:33.168397 2074 tablet_replica.cc:331] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:33.168569 2074 raft_consensus.cc:2241] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:33.168678 2074 raft_consensus.cc:2270] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:33.169128 1478 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Sending DeleteTablet for 3 replicas of tablet 703110ded5c348cb96e47b437f340375
I20250626 14:04:33.169430 1478 catalog_manager.cc:6652] ReplaceTablet: tablet 703110ded5c348cb96e47b437f340375 deleted and replaced by tablet 6a0b30a82a024246a45dc64dce32947d
I20250626 14:04:33.170300 1658 tablet_service.cc:1515] Processing DeleteTablet for tablet 703110ded5c348cb96e47b437f340375 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a0b30a82a024246a45dc64dce32947d) from {username='slave'} at 127.0.0.1:43706
I20250626 14:04:33.170495 1792 tablet_service.cc:1515] Processing DeleteTablet for tablet 703110ded5c348cb96e47b437f340375 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a0b30a82a024246a45dc64dce32947d) from {username='slave'} at 127.0.0.1:48162
I20250626 14:04:33.170975 1927 tablet_service.cc:1515] Processing DeleteTablet for tablet 703110ded5c348cb96e47b437f340375 with delete_type TABLET_DATA_DELETED (replaced by tablet 6a0b30a82a024246a45dc64dce32947d) from {username='slave'} at 127.0.0.1:54640
I20250626 14:04:33.171008 2075 tablet_replica.cc:331] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:33.171144 2075 raft_consensus.cc:2241] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:33.171244 2075 raft_consensus.cc:2270] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:33.171599 2075 ts_tablet_manager.cc:1905] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:33.171991 1478 master_service.cc:948] ReplaceTablet: received request to replace tablet 34cd6172930d497698cb9239220566af from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:33.172355 1478 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_2c8dee0163784a60a8468b53619247f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:33.172889 2073 tablet_replica.cc:331] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:33.173031 2073 raft_consensus.cc:2241] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:33.173285 2073 pending_rounds.cc:70] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b: Trying to abort 1 pending ops.
I20250626 14:04:33.173398 2073 pending_rounds.cc:77] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171877571629690880 op_type: NO_OP noop_request { }
I20250626 14:04:33.173517 2073 raft_consensus.cc:2887] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 14:04:33.173595 2073 raft_consensus.cc:2270] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:33.173880 2059 tablet_bootstrap.cc:492] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:33.173988 2059 ts_tablet_manager.cc:1397] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.002s
I20250626 14:04:33.174512 2059 raft_consensus.cc:357] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.174614 2059 raft_consensus.cc:383] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.174643 2059 raft_consensus.cc:738] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.175274 2074 ts_tablet_manager.cc:1905] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:33.177950 1792 tablet_service.cc:1468] Processing CreateTablet for tablet 6a0b30a82a024246a45dc64dce32947d (DEFAULT_TABLE table=default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.178237 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a0b30a82a024246a45dc64dce32947d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.178905 2073 ts_tablet_manager.cc:1905] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:33.179076 2059 consensus_queue.cc:260] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.181923 2059 ts_tablet_manager.cc:1428] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20250626 14:04:33.183413 2060 tablet_bootstrap.cc:492] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:33.184480 2060 tablet_bootstrap.cc:654] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.185690 1658 tablet_service.cc:1468] Processing CreateTablet for tablet 6a0b30a82a024246a45dc64dce32947d (DEFAULT_TABLE table=default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.185953 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a0b30a82a024246a45dc64dce32947d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.188109 2061 tablet_bootstrap.cc:492] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:33.189189 2061 tablet_bootstrap.cc:654] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.194914 2075 ts_tablet_manager.cc:1918] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 14:04:33.195073 2075 log.cc:1199] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/703110ded5c348cb96e47b437f340375
I20250626 14:04:33.195384 2075 ts_tablet_manager.cc:1939] T 703110ded5c348cb96e47b437f340375 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:33.196259 1472 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet 703110ded5c348cb96e47b437f340375 (table default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81]) successfully deleted
I20250626 14:04:33.183211 1927 tablet_service.cc:1468] Processing CreateTablet for tablet 6a0b30a82a024246a45dc64dce32947d (DEFAULT_TABLE table=default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:33.192822 2083 raft_consensus.cc:491] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:33.197000 2083 raft_consensus.cc:513] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.197151 1927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a0b30a82a024246a45dc64dce32947d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:33.197420 2083 leader_election.cc:290] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:33.197942 1812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd5f0eb6de154b3c9ffdd6d8f7042a49" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:33.198060 1812 raft_consensus.cc:2466] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:33.198351 1883 leader_election.cc:304] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:33.198726 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd5f0eb6de154b3c9ffdd6d8f7042a49" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:33.198851 1677 raft_consensus.cc:2466] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:33.199012 2083 raft_consensus.cc:2802] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:33.199060 2083 raft_consensus.cc:491] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:33.199095 2083 raft_consensus.cc:3058] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.199959 2083 raft_consensus.cc:513] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.200300 2083 leader_election.cc:290] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Requested vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:33.200874 1812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd5f0eb6de154b3c9ffdd6d8f7042a49" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:33.200985 1812 raft_consensus.cc:3058] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.201920 1812 raft_consensus.cc:2466] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:33.202298 1883 leader_election.cc:304] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:33.202817 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd5f0eb6de154b3c9ffdd6d8f7042a49" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:33.202937 1677 raft_consensus.cc:3058] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.203940 1677 raft_consensus.cc:2466] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:33.204869 2083 raft_consensus.cc:2802] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:33.204967 2083 raft_consensus.cc:695] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Running, Role: LEADER
I20250626 14:04:33.205058 2068 raft_consensus.cc:491] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:33.205113 2083 consensus_queue.cc:237] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.205145 2068 raft_consensus.cc:513] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.205566 2068 leader_election.cc:290] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:33.208343 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ace98628290843938dc5db70e5a43a37" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:33.208469 1947 raft_consensus.cc:2466] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 0.
I20250626 14:04:33.208990 1613 leader_election.cc:304] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:33.209172 2068 raft_consensus.cc:2802] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:33.209239 2068 raft_consensus.cc:491] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:33.209285 2068 raft_consensus.cc:3058] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.210341 2068 raft_consensus.cc:513] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.210757 2068 leader_election.cc:290] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 election: Requested vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:33.211311 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ace98628290843938dc5db70e5a43a37" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:33.211428 1813 raft_consensus.cc:3058] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.211642 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ace98628290843938dc5db70e5a43a37" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:33.211777 1947 raft_consensus.cc:3058] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.212549 1813 raft_consensus.cc:2466] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
I20250626 14:04:33.212795 1947 raft_consensus.cc:2466] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
I20250626 14:04:33.213585 1613 leader_election.cc:304] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:33.214068 1812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ace98628290843938dc5db70e5a43a37" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:33.214205 1812 raft_consensus.cc:2374] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1. Re-sending same reply.
I20250626 14:04:33.214530 2068 raft_consensus.cc:2802] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:33.214648 2068 raft_consensus.cc:695] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [term 1 LEADER]: Becoming Leader. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Running, Role: LEADER
I20250626 14:04:33.214800 2068 consensus_queue.cc:237] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.222932 2059 tablet_bootstrap.cc:492] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:33.224185 2059 tablet_bootstrap.cc:654] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.227018 2099 raft_consensus.cc:491] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:33.227133 2099 raft_consensus.cc:513] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.228550 2099 leader_election.cc:290] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:33.228904 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7fa3619f0c541bf8e83db2d4fd24ced" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:33.229035 1677 raft_consensus.cc:2466] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:33.229372 1749 leader_election.cc:304] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:33.229955 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7fa3619f0c541bf8e83db2d4fd24ced" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:33.230087 1947 raft_consensus.cc:2466] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:33.230815 2099 raft_consensus.cc:2802] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:33.230875 2099 raft_consensus.cc:491] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:33.230912 2099 raft_consensus.cc:3058] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.231909 2099 raft_consensus.cc:513] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.232764 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7fa3619f0c541bf8e83db2d4fd24ced" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:33.232895 1677 raft_consensus.cc:3058] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.233887 1677 raft_consensus.cc:2466] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:33.234231 1749 leader_election.cc:304] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:33.234443 2209 raft_consensus.cc:2802] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:33.234565 2209 raft_consensus.cc:695] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Becoming Leader. State: Replica: e517893e13f546c0b40d976633b49651, State: Running, Role: LEADER
I20250626 14:04:33.234733 2209 consensus_queue.cc:237] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.234992 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7fa3619f0c541bf8e83db2d4fd24ced" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:33.235113 1947 raft_consensus.cc:3058] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.236147 1947 raft_consensus.cc:2466] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:33.236907 2099 leader_election.cc:290] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Requested vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:33.240470 2074 ts_tablet_manager.cc:1918] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 14:04:33.241268 2086 consensus_queue.cc:1035] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.241436 2074 log.cc:1199] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/34cd6172930d497698cb9239220566af
I20250626 14:04:33.241665 2074 ts_tablet_manager.cc:1939] T 34cd6172930d497698cb9239220566af P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:33.243031 2073 ts_tablet_manager.cc:1918] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:33.243419 1472 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet 34cd6172930d497698cb9239220566af (table default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738]) successfully deleted
I20250626 14:04:33.243795 2074 tablet_replica.cc:331] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:33.243973 2074 raft_consensus.cc:2241] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:33.244292 2074 pending_rounds.cc:70] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99: Trying to abort 1 pending ops.
I20250626 14:04:33.244426 2073 log.cc:1199] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/34cd6172930d497698cb9239220566af
I20250626 14:04:33.244760 2073 ts_tablet_manager.cc:1939] T 34cd6172930d497698cb9239220566af P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:33.245538 1473 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet 34cd6172930d497698cb9239220566af (table default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738]) successfully deleted
I20250626 14:04:33.245863 2073 tablet_replica.cc:331] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:33.245993 2073 raft_consensus.cc:2241] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:33.246076 2073 raft_consensus.cc:2270] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:33.246409 2073 ts_tablet_manager.cc:1905] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:33.247792 2083 consensus_queue.cc:1035] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.250064 2074 pending_rounds.cc:77] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171877571628765184 op_type: NO_OP noop_request { }
I20250626 14:04:33.250207 2074 raft_consensus.cc:2887] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 14:04:33.250358 2074 raft_consensus.cc:2270] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:33.251356 2186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81db61eed5584e7e8b0cf40e5441f88e. Latest consensus state: current_term: 1 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:33.251482 2186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:33.251961 2065 raft_consensus.cc:491] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:33.252053 2065 raft_consensus.cc:513] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.252439 2065 leader_election.cc:290] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:33.253124 1812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f51154f6fd584a95976d6af90f9133e9" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:33.253259 1812 raft_consensus.cc:2466] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:33.253543 1883 leader_election.cc:304] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:33.253736 2065 raft_consensus.cc:2802] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:33.253784 2065 raft_consensus.cc:491] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:33.253822 2065 raft_consensus.cc:3058] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.254892 2065 raft_consensus.cc:513] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.254925 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f51154f6fd584a95976d6af90f9133e9" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:33.255043 1678 raft_consensus.cc:2466] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:33.255229 2065 leader_election.cc:290] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Requested vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:33.255905 1812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f51154f6fd584a95976d6af90f9133e9" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:33.256027 1812 raft_consensus.cc:3058] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.256176 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f51154f6fd584a95976d6af90f9133e9" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:33.256283 1678 raft_consensus.cc:3058] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.257207 1812 raft_consensus.cc:2466] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:33.257349 1678 raft_consensus.cc:2466] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:33.257484 1883 leader_election.cc:304] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:33.257771 2065 raft_consensus.cc:2802] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:33.257861 2065 raft_consensus.cc:695] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Running, Role: LEADER
I20250626 14:04:33.257993 2065 consensus_queue.cc:237] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.248811 2059 tablet_bootstrap.cc:492] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:33.260653 2059 ts_tablet_manager.cc:1397] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.001s
I20250626 14:04:33.261289 2059 raft_consensus.cc:357] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.261509 2059 raft_consensus.cc:383] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.261602 2059 raft_consensus.cc:738] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.261834 2059 consensus_queue.cc:260] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.263574 2059 ts_tablet_manager.cc:1428] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250626 14:04:33.264997 2074 ts_tablet_manager.cc:1905] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:33.265120 2059 tablet_bootstrap.cc:492] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:33.266289 2059 tablet_bootstrap.cc:654] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:33.267869 2074 ts_tablet_manager.cc:1918] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:33.267948 2074 log.cc:1199] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/703110ded5c348cb96e47b437f340375
I20250626 14:04:33.268194 2074 ts_tablet_manager.cc:1939] T 703110ded5c348cb96e47b437f340375 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:33.269114 2059 tablet_bootstrap.cc:492] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:33.269258 2059 ts_tablet_manager.cc:1397] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250626 14:04:33.269812 2059 raft_consensus.cc:357] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.269980 2059 raft_consensus.cc:383] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.270053 2059 raft_consensus.cc:738] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.270159 2073 ts_tablet_manager.cc:1918] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 14:04:33.270249 2059 consensus_queue.cc:260] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.271057 2060 tablet_bootstrap.cc:492] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:33.271148 2060 ts_tablet_manager.cc:1397] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.088s user 0.002s sys 0.000s
I20250626 14:04:33.271591 2060 raft_consensus.cc:357] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.271699 2060 raft_consensus.cc:383] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.271731 2060 raft_consensus.cc:738] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.271860 2060 consensus_queue.cc:260] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.272121 2060 ts_tablet_manager.cc:1428] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:33.272972 1472 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet 703110ded5c348cb96e47b437f340375 (table default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81]) successfully deleted
I20250626 14:04:33.273774 2059 ts_tablet_manager.cc:1428] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250626 14:04:33.275039 2073 log.cc:1199] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/703110ded5c348cb96e47b437f340375
I20250626 14:04:33.275383 2073 ts_tablet_manager.cc:1939] T 703110ded5c348cb96e47b437f340375 P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:33.276340 1473 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet 703110ded5c348cb96e47b437f340375 (table default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81]) successfully deleted
I20250626 14:04:33.277000 2061 tablet_bootstrap.cc:492] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:33.277091 2061 ts_tablet_manager.cc:1397] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.000s
I20250626 14:04:33.277565 2061 raft_consensus.cc:357] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.277669 2061 raft_consensus.cc:383] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:33.277700 2061 raft_consensus.cc:738] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:33.277822 2061 consensus_queue.cc:260] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.278076 2061 ts_tablet_manager.cc:1428] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:33.283200 2099 consensus_queue.cc:1035] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.297799 2209 consensus_queue.cc:1035] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.298923 2083 raft_consensus.cc:491] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:33.299041 2083 raft_consensus.cc:513] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.299504 2083 leader_election.cc:290] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:33.299955 1812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "071dcb923e314fca9fbbf94592b01fe2" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:33.300093 1812 raft_consensus.cc:2466] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:33.300631 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "071dcb923e314fca9fbbf94592b01fe2" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:33.300693 1883 leader_election.cc:304] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:33.300765 1677 raft_consensus.cc:2466] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:33.302074 2086 raft_consensus.cc:2802] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:33.302135 2086 raft_consensus.cc:491] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:33.302176 2086 raft_consensus.cc:3058] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.303218 2086 raft_consensus.cc:513] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.303586 2086 leader_election.cc:290] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Requested vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:33.303884 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "071dcb923e314fca9fbbf94592b01fe2" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:33.304021 1677 raft_consensus.cc:3058] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.304967 1677 raft_consensus.cc:2466] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:33.305372 1884 leader_election.cc:304] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:33.305431 1812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "071dcb923e314fca9fbbf94592b01fe2" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:33.305528 2065 raft_consensus.cc:2802] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:33.305553 1812 raft_consensus.cc:3058] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.305610 2065 raft_consensus.cc:695] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Running, Role: LEADER
I20250626 14:04:33.305745 2065 consensus_queue.cc:237] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.306489 1812 raft_consensus.cc:2466] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:33.311753 2099 raft_consensus.cc:491] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:33.311862 2099 raft_consensus.cc:513] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.312228 2099 leader_election.cc:290] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:33.312937 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6a0b30a82a024246a45dc64dce32947d" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:33.313074 1677 raft_consensus.cc:2466] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:33.313383 1749 leader_election.cc:304] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:33.313604 2099 raft_consensus.cc:2802] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:33.313655 2099 raft_consensus.cc:491] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:33.313701 2099 raft_consensus.cc:3058] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.314594 2099 raft_consensus.cc:513] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.314924 2099 leader_election.cc:290] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Requested vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:33.315682 1948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6a0b30a82a024246a45dc64dce32947d" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:33.315793 1948 raft_consensus.cc:3058] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.316826 1948 raft_consensus.cc:2466] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:33.317229 1748 leader_election.cc:304] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:33.317493 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6a0b30a82a024246a45dc64dce32947d" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:33.317595 1947 raft_consensus.cc:2374] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1. Re-sending same reply.
I20250626 14:04:33.318228 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6a0b30a82a024246a45dc64dce32947d" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:33.318352 1677 raft_consensus.cc:3058] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.319305 1677 raft_consensus.cc:2466] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:33.319589 2099 raft_consensus.cc:2802] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:33.319676 2099 raft_consensus.cc:695] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Becoming Leader. State: Replica: e517893e13f546c0b40d976633b49651, State: Running, Role: LEADER
I20250626 14:04:33.319851 2099 consensus_queue.cc:237] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.440666 2099 raft_consensus.cc:491] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:33.440820 2099 raft_consensus.cc:513] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.441316 2099 leader_election.cc:290] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:33.441792 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0f427caf3744354ae36c40f257ec89c" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:33.441792 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0f427caf3744354ae36c40f257ec89c" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:33.441949 1947 raft_consensus.cc:2466] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:33.441949 1677 raft_consensus.cc:2466] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:33.442320 1749 leader_election.cc:304] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:33.442584 2099 raft_consensus.cc:2802] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:33.442637 2099 raft_consensus.cc:491] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:33.442690 2099 raft_consensus.cc:3058] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.443490 2099 raft_consensus.cc:513] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.443879 2099 leader_election.cc:290] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Requested vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:33.444285 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0f427caf3744354ae36c40f257ec89c" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:33.444406 1947 raft_consensus.cc:3058] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.445410 1947 raft_consensus.cc:2466] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:33.445829 1748 leader_election.cc:304] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:33.446257 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0f427caf3744354ae36c40f257ec89c" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:33.446384 1677 raft_consensus.cc:3058] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:33.447352 1677 raft_consensus.cc:2466] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:33.447976 2099 raft_consensus.cc:2802] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:33.448068 2099 raft_consensus.cc:695] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Becoming Leader. State: Replica: e517893e13f546c0b40d976633b49651, State: Running, Role: LEADER
I20250626 14:04:33.448204 2099 consensus_queue.cc:237] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:33.484491 1484 catalog_manager.cc:3460] renamed table default.table_1e68626ebae94b208424f5f5a66dd06e [id=aa8a6a6f408d454abdc5948e21f84233] in HMS: new name default.table_afd26acd01c7422fbb06520af5a4f76c
I20250626 14:04:33.581596 2209 consensus_queue.cc:1035] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.581923 2209 consensus_queue.cc:1035] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.618158 2099 consensus_queue.cc:1035] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.630533 2069 consensus_queue.cc:1035] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.637827 2068 consensus_queue.cc:1035] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.657920 2083 consensus_queue.cc:1035] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.665591 2066 consensus_queue.cc:1035] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.667776 2065 consensus_queue.cc:1035] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.682909 2252 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81db61eed5584e7e8b0cf40e5441f88e)
I20250626 14:04:33.683043 2252 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:33.687238 2252 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
I20250626 14:04:33.687543 1496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24ad911d71b6433f8c7939a1d1f7b782" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "81db61eed5584e7e8b0cf40e5441f88e" is_pre_election: true
I20250626 14:04:33.689330 2186 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81db61eed5584e7e8b0cf40e5441f88e)
I20250626 14:04:33.689529 2186 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:33.690860 2186 leader_election.cc:290] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
I20250626 14:04:33.691642 1354 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07532b7648e4414789ed2b099bad2b19" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "24ad911d71b6433f8c7939a1d1f7b782" is_pre_election: true
I20250626 14:04:33.691974 1496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07532b7648e4414789ed2b099bad2b19" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "81db61eed5584e7e8b0cf40e5441f88e" is_pre_election: true
I20250626 14:04:33.692200 2149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24ad911d71b6433f8c7939a1d1f7b782" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "07532b7648e4414789ed2b099bad2b19" is_pre_election: true
I20250626 14:04:33.692435 2149 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ad911d71b6433f8c7939a1d1f7b782 in term 1.
I20250626 14:04:33.692552 2124 leader_election.cc:304] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19; no voters: 24ad911d71b6433f8c7939a1d1f7b782, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:04:33.693049 1330 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782; no voters: 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:04:33.693495 2186 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 14:04:33.693744 2252 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 14:04:33.693859 2252 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 81db61eed5584e7e8b0cf40e5441f88e)
I20250626 14:04:33.693967 2252 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 1 FOLLOWER]: Advancing to term 2
I20250626 14:04:33.695094 2252 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:33.695793 2149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24ad911d71b6433f8c7939a1d1f7b782" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "07532b7648e4414789ed2b099bad2b19"
I20250626 14:04:33.695904 2149 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 1 FOLLOWER]: Advancing to term 2
I20250626 14:04:33.696048 2252 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 2 election: Requested vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
I20250626 14:04:33.696092 1496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24ad911d71b6433f8c7939a1d1f7b782" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "81db61eed5584e7e8b0cf40e5441f88e"
I20250626 14:04:33.697877 2149 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ad911d71b6433f8c7939a1d1f7b782 in term 2.
I20250626 14:04:33.698403 1330 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782; no voters: 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:04:33.698640 2252 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 14:04:33.698825 2252 raft_consensus.cc:695] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 2 LEADER]: Becoming Leader. State: Replica: 24ad911d71b6433f8c7939a1d1f7b782, State: Running, Role: LEADER
I20250626 14:04:33.699127 2252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 39, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:33.699798 2065 consensus_queue.cc:1035] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.699994 2252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24ad911d71b6433f8c7939a1d1f7b782. Latest consensus state: current_term: 2 leader_uuid: "24ad911d71b6433f8c7939a1d1f7b782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:33.700078 2252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:33.711812 2083 consensus_queue.cc:1035] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.716789 2256 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 14:04:33.718837 2256 catalog_manager.cc:671] Loaded metadata for table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]
I20250626 14:04:33.719257 2256 catalog_manager.cc:671] Loaded metadata for table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]
I20250626 14:04:33.719661 2256 catalog_manager.cc:671] Loaded metadata for table default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738]
I20250626 14:04:33.720860 2256 catalog_manager.cc:671] Loaded metadata for table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9]
I20250626 14:04:33.721140 2256 catalog_manager.cc:671] Loaded metadata for table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]
I20250626 14:04:33.721379 2256 catalog_manager.cc:671] Loaded metadata for table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]
I20250626 14:04:33.721632 2256 catalog_manager.cc:671] Loaded metadata for table default.table_1e68626ebae94b208424f5f5a66dd06e [id=aa8a6a6f408d454abdc5948e21f84233]
I20250626 14:04:33.722010 2256 catalog_manager.cc:671] Loaded metadata for table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]
I20250626 14:04:33.722266 2256 catalog_manager.cc:671] Loaded metadata for table default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81]
I20250626 14:04:33.722513 2256 catalog_manager.cc:671] Loaded metadata for table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]
I20250626 14:04:33.723945 2256 tablet_loader.cc:96] loaded metadata for tablet 071dcb923e314fca9fbbf94592b01fe2 (table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a])
I20250626 14:04:33.724344 2256 tablet_loader.cc:96] loaded metadata for tablet 5f5ce7fc87164ae1ace91557275285f8 (table default.table_1e68626ebae94b208424f5f5a66dd06e [id=aa8a6a6f408d454abdc5948e21f84233])
I20250626 14:04:33.724613 2256 tablet_loader.cc:96] loaded metadata for tablet 680301fb5722406194f4dcadeb726964 (table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9])
I20250626 14:04:33.724833 2256 tablet_loader.cc:96] loaded metadata for tablet 6a0b30a82a024246a45dc64dce32947d (table default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81])
I20250626 14:04:33.725271 2256 tablet_loader.cc:96] loaded metadata for tablet ace98628290843938dc5db70e5a43a37 (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140])
I20250626 14:04:33.725493 2256 tablet_loader.cc:96] loaded metadata for tablet d0f427caf3744354ae36c40f257ec89c (table default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738])
I20250626 14:04:33.725695 2256 tablet_loader.cc:96] loaded metadata for tablet d2a95d2f060943a68721021a204e0cc3 (table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75])
I20250626 14:04:33.725889 2256 tablet_loader.cc:96] loaded metadata for tablet dd5f0eb6de154b3c9ffdd6d8f7042a49 (table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03])
I20250626 14:04:33.726064 2256 tablet_loader.cc:96] loaded metadata for tablet f51154f6fd584a95976d6af90f9133e9 (table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80])
I20250626 14:04:33.727173 2256 tablet_loader.cc:96] loaded metadata for tablet f7fa3619f0c541bf8e83db2d4fd24ced (table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447])
I20250626 14:04:33.727476 2256 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 14:04:33.728013 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:33.728154 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:33.728161 2256 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:33.728206 2256 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 14:04:33.728894 2149 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 2 FOLLOWER]: Rejecting Update request from peer 81db61eed5584e7e8b0cf40e5441f88e for earlier term 1. Current term is 2. Ops: []
I20250626 14:04:33.729019 2256 catalog_manager.cc:1506] Loading token signing keys...
I20250626 14:04:33.729413 1564 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 }, Status: INVALID_TERM, Last received: 1.39, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20250626 14:04:33.729589 1564 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 1 LEADER]: Stepping down as leader of term 1
I20250626 14:04:33.729626 2256 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Loaded TSK: 0
I20250626 14:04:33.729641 1564 raft_consensus.cc:738] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 81db61eed5584e7e8b0cf40e5441f88e, State: Running, Role: LEADER
I20250626 14:04:33.729830 1564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 39, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:33.730103 1564 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 1 FOLLOWER]: Advancing to term 2
I20250626 14:04:33.730629 2256 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 14:04:33.731190 2256 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250626 14:04:33.731827 2256 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 10
W20250626 14:04:33.732120 2260 catalog_manager.cc:1411] Failed waiting for node to catch up after master election: Illegal state: Node 81db61eed5584e7e8b0cf40e5441f88e not leader. Raft Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:33.843780 2209 consensus_queue.cc:1035] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.844997 1995 heartbeater.cc:499] Master 127.1.11.254:36707 was elected leader, sending a full tablet report...
I20250626 14:04:33.847203 1344 catalog_manager.cc:5582] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195). New cstate: current_term: 1 leader_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250626 14:04:33.847805 1860 heartbeater.cc:499] Master 127.1.11.254:36707 was elected leader, sending a full tablet report...
I20250626 14:04:33.847855 1344 catalog_manager.cc:5582] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193). New cstate: current_term: 1 leader_uuid: "4e57adb83e9f451488ed1d0034043e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } }
I20250626 14:04:33.848035 1344 catalog_manager.cc:5582] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99 reported cstate change: term changed from 0 to 1, leader changed from <none> to e517893e13f546c0b40d976633b49651 (127.1.11.194). New cstate: current_term: 1 leader_uuid: "e517893e13f546c0b40d976633b49651" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } }
I20250626 14:04:33.848186 1344 catalog_manager.cc:5582] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195). New cstate: current_term: 1 leader_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250626 14:04:33.848364 1344 catalog_manager.cc:5582] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195). New cstate: current_term: 1 leader_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } health_report { overall_health: HEALTHY } } }
I20250626 14:04:33.849720 1344 catalog_manager.cc:5582] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99 reported cstate change: term changed from 0 to 1, leader changed from <none> to e517893e13f546c0b40d976633b49651 (127.1.11.194). New cstate: current_term: 1 leader_uuid: "e517893e13f546c0b40d976633b49651" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } }
I20250626 14:04:33.850594 2069 consensus_queue.cc:1035] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.857002 2149 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 2 FOLLOWER]: Refusing update from remote peer 24ad911d71b6433f8c7939a1d1f7b782: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 2 index: 41. (index mismatch)
I20250626 14:04:33.857101 1496 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 2 FOLLOWER]: Refusing update from remote peer 24ad911d71b6433f8c7939a1d1f7b782: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 2 index: 41. (index mismatch)
I20250626 14:04:33.857757 2255 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20250626 14:04:33.858541 2255 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20250626 14:04:33.859815 2186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24ad911d71b6433f8c7939a1d1f7b782. Latest consensus state: current_term: 2 leader_uuid: "24ad911d71b6433f8c7939a1d1f7b782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:33.859915 2186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:33.861457 2259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24ad911d71b6433f8c7939a1d1f7b782. Latest consensus state: current_term: 2 leader_uuid: "24ad911d71b6433f8c7939a1d1f7b782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:33.861562 2259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:33.864883 1343 catalog_manager.cc:5582] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 reported cstate change: term changed from 0 to 1, leader changed from <none> to e517893e13f546c0b40d976633b49651 (127.1.11.194). New cstate: current_term: 1 leader_uuid: "e517893e13f546c0b40d976633b49651" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20250626 14:04:33.866247 2259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "24ad911d71b6433f8c7939a1d1f7b782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:33.866371 2259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:33.867362 2186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "24ad911d71b6433f8c7939a1d1f7b782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:33.867484 2186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:33.868109 2252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "24ad911d71b6433f8c7939a1d1f7b782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:33.868202 2252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:33.868145 2266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "24ad911d71b6433f8c7939a1d1f7b782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:33.868243 2266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:33.877266 2264 mvcc.cc:204] Tried to move back new op lower bound from 7171877576113045504 to 7171877575474114560. Current Snapshot: MvccSnapshot[applied={T|T < 7171877576113045504}]
I20250626 14:04:33.883623 2065 consensus_queue.cc:1035] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.884917 1725 heartbeater.cc:499] Master 127.1.11.254:36707 was elected leader, sending a full tablet report...
I20250626 14:04:33.890453 2086 consensus_queue.cc:1035] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.891455 2099 consensus_queue.cc:1035] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.897648 2099 consensus_queue.cc:1035] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:33.901877 1485 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81]
I20250626 14:04:33.941447 1479 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738]
W20250626 14:04:33.964841 1560 catalog_manager.cc:3897] Illegal state: an error occurred while updating the sys-catalog: replica 81db61eed5584e7e8b0cf40e5441f88e is not leader of this config: current role FOLLOWER
W20250626 14:04:33.965068 1560 hms_notification_log_listener.cc:329] Failed to handle Hive Metastore notification: 11 ALTER_TABLE default.table_afd26acd01c7422fbb06520af5a4f76c: Illegal state: an error occurred while updating the sys-catalog: replica 81db61eed5584e7e8b0cf40e5441f88e is not leader of this config: current role FOLLOWER
W20250626 14:04:33.965153 1560 hms_notification_log_listener.cc:143] Hive Metastore notification log listener poll failed: Service unavailable: lost leadership while handling Hive Metastore notification log events: an error occurred while updating the sys-catalog: replica 81db61eed5584e7e8b0cf40e5441f88e is not leader of this config: current role FOLLOWER
I20250626 14:04:33.965348 1481 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51198 (request call id 23) took 1090 ms. Trace:
I20250626 14:04:33.965471 1483 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51198 (request call id 20) took 1199 ms. Trace:
I20250626 14:04:33.965533 1483 rpcz_store.cc:276] 0626 14:04:32.766287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 14:04:32.790675 (+ 24388us) service_pool.cc:225] Handling call
0626 14:04:33.965462 (+1174787us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 14:04:33.966118 1483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.11.195:54487 (request call id 13) took 784 ms (client timeout 1000 ms). Trace:
W20250626 14:04:33.966171 1483 rpcz_store.cc:269] 0626 14:04:33.181296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 14:04:33.965692 (+784396us) service_pool.cc:225] Handling call
0626 14:04:33.966092 (+ 400us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 14:04:33.966403 2001 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.11.252:37925)
W20250626 14:04:33.966701 1483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.11.194:56877 (request call id 16) took 775 ms (client timeout 1000 ms). Trace:
W20250626 14:04:33.966763 1483 rpcz_store.cc:269] 0626 14:04:33.191292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 14:04:33.966305 (+775013us) service_pool.cc:225] Handling call
0626 14:04:33.966688 (+ 383us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 14:04:33.967386 1483 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51198 (request call id 31) took 1037 ms. Trace:
I20250626 14:04:33.967432 1483 rpcz_store.cc:276] 0626 14:04:32.929549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 14:04:33.967258 (+1037709us) service_pool.cc:225] Handling call
0626 14:04:33.967375 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 14:04:33.968621 1482 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51198 (request call id 21) took 1175 ms. Trace:
I20250626 14:04:33.968684 1482 rpcz_store.cc:276] 0626 14:04:32.792874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 14:04:32.943780 (+150906us) service_pool.cc:225] Handling call
0626 14:04:33.968609 (+1024829us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 14:04:33.968943 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet ace98628290843938dc5db70e5a43a37 from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:33.969123 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet 071dcb923e314fca9fbbf94592b01fe2 from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:33.969280 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet ace98628290843938dc5db70e5a43a37 from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:33.969434 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet d2a95d2f060943a68721021a204e0cc3 from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:33.969595 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet 6a0b30a82a024246a45dc64dce32947d from {username='slave'} at 127.0.0.1:51198
W20250626 14:04:33.970739 2001 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.11.252:37925)
I20250626 14:04:33.970774 1486 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51198 (request call id 22) took 1153 ms. Trace:
I20250626 14:04:33.970824 1486 rpcz_store.cc:276] 0626 14:04:32.817863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 14:04:32.950022 (+132159us) service_pool.cc:225] Handling call
0626 14:04:33.970765 (+1020743us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 14:04:33.970959 1480 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51198 (request call id 24) took 1085 ms. Trace:
I20250626 14:04:33.971000 1480 rpcz_store.cc:276] 0626 14:04:32.885828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 14:04:33.029157 (+143329us) service_pool.cc:225] Handling call
0626 14:04:33.970949 (+941792us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 14:04:33.971199 2001 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.11.252:37925)
W20250626 14:04:33.971396 1478 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2c8dee0163784a60a8468b53619247f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 14:04:33.971956 2001 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.11.252:37925)
I20250626 14:04:33.972342 2080 master_runner.cc:232] Existing masters have differing terms: 1 vs 2
I20250626 14:04:33.972460 2080 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:04:33.972523 1484 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51198 (request call id 11) took 1406 ms. Trace:
I20250626 14:04:33.972601 1484 rpcz_store.cc:276] 0626 14:04:32.566329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 14:04:32.566368 (+ 39us) service_pool.cc:225] Handling call
0626 14:04:32.740187 (+173819us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 14:04:33.972048 (+1231861us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 14:04:33.972307 (+ 259us) catalog_manager.cc:3683] Apply alter schema
0626 14:04:33.972511 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":360409,"Hive Metastore.run_cpu_time_us":812,"Hive Metastore.run_wall_time_us":383659}
I20250626 14:04:33.972779 1485 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51198 (request call id 12) took 1355 ms. Trace:
I20250626 14:04:33.972821 1485 rpcz_store.cc:276] 0626 14:04:32.617045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 14:04:32.617096 (+ 51us) service_pool.cc:225] Handling call
0626 14:04:32.740397 (+123301us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 14:04:33.972770 (+1232373us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":743970,"Hive Metastore.run_cpu_time_us":225,"Hive Metastore.run_wall_time_us":417212}
I20250626 14:04:33.972970 1479 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51198 (request call id 18) took 1298 ms. Trace:
I20250626 14:04:33.973009 1479 rpcz_store.cc:276] 0626 14:04:32.674783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 14:04:32.699746 (+ 24963us) service_pool.cc:225] Handling call
0626 14:04:32.747523 (+ 47777us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 14:04:33.972961 (+1225438us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":1154077,"Hive Metastore.run_cpu_time_us":199,"Hive Metastore.run_wall_time_us":39652}
I20250626 14:04:33.965436 1481 rpcz_store.cc:276] 0626 14:04:32.874723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 14:04:32.981588 (+106865us) service_pool.cc:225] Handling call
0626 14:04:33.965327 (+983739us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 14:04:33.979377 1342 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59552:
table { table_name: "default.Table_fb8ebc7aa8894f28b4d79252382f356a" } new_table_name: "default.Table_7d5a9b0c3ae440b89dac3ca5dd8e81ef" modify_external_catalogs: true
I20250626 14:04:33.979594 1343 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59552:
name: "default.Table_0b500f44e16e45e589c01ab7ad9a621b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:33.979704 1344 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59552:
table { table_name: "default.Table_e6e7d4c291af4d29baf43e204c5292f0" } new_table_name: "default.Table_f01c05221e5c4e75bf700b9474072881" modify_external_catalogs: true
I20250626 14:04:33.979841 1341 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59552:
name: "default.Table_f98cc740eae3444b88bb688eb8c8a316"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:33.979920 1336 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59552:
name: "default.Table_42d07c2b16314b1999eea5f3878944fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:33.980130 1340 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59552:
table { table_name: "default.Table_0be8dc44c93040319885961d6e4a0608" } modify_external_catalogs: true
I20250626 14:04:33.980219 1340 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59552:
table { table_name: "default.Table_0be8dc44c93040319885961d6e4a0608" } modify_external_catalogs: true
I20250626 14:04:33.980392 1339 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59552:
name: "default.Table_34cb28e2c8d740aba5c52377533fdfa4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:33.980634 1338 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59552:
table { table_name: "default.Table_f9b5b5e2dcf64f9e889c618242274d25" } modify_external_catalogs: true
I20250626 14:04:33.980726 1338 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59552:
table { table_name: "default.Table_f9b5b5e2dcf64f9e889c618242274d25" } modify_external_catalogs: true
I20250626 14:04:33.980875 1335 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59552:
table { table_name: "default.Table_3a3f21b3c0e54e438561e02ab79fa36b" } new_table_name: "default.Table_3633fc38c5374180a081fab63a1eed84" modify_external_catalogs: true
I20250626 14:04:33.979377 1337 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59552:
name: "default.Table_5bf40f09daaa4240a8bcab30e1aa341f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:33.998498 1542 catalog_manager.cc:2747] Deleting table default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81] in response to Hive Metastore notification log event 12
I20250626 14:04:34.008517 1542 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Sending DeleteTablet for 3 replicas of tablet 6a0b30a82a024246a45dc64dce32947d
I20250626 14:04:34.012122 1542 catalog_manager.cc:2747] Deleting table default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738] in response to Hive Metastore notification log event 13
I20250626 14:04:34.020641 1928 tablet_service.cc:1515] Processing DeleteTablet for tablet 6a0b30a82a024246a45dc64dce32947d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 14:04:33 UTC) from {username='slave'} at 127.0.0.1:54652
I20250626 14:04:34.021193 1542 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Sending DeleteTablet for 3 replicas of tablet d0f427caf3744354ae36c40f257ec89c
I20250626 14:04:34.021937 1928 tablet_service.cc:1515] Processing DeleteTablet for tablet d0f427caf3744354ae36c40f257ec89c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 14:04:34 UTC) from {username='slave'} at 127.0.0.1:54652
W20250626 14:04:34.022858 1336 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_42d07c2b16314b1999eea5f3878944fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250626 14:04:34.023018 1341 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f98cc740eae3444b88bb688eb8c8a316 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 14:04:34.024186 2282 tablet_replica.cc:331] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:34.024343 2282 raft_consensus.cc:2241] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:34.024494 2282 raft_consensus.cc:2270] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 14:04:34.022858 1343 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0b500f44e16e45e589c01ab7ad9a621b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 14:04:34.025134 2282 ts_tablet_manager.cc:1905] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:34.025768 1477 catalog_manager.cc:2706] deleted HMS catalog entry for table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9]
I20250626 14:04:34.027025 1477 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51198 (request call id 19) took 1318 ms. Trace:
I20250626 14:04:34.027096 1477 rpcz_store.cc:276] 0626 14:04:32.709037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 14:04:32.709079 (+ 42us) service_pool.cc:225] Handling call
0626 14:04:33.122782 (+413703us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0626 14:04:34.027005 (+904223us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":839972,"Hive Metastore.run_cpu_time_us":122,"Hive Metastore.run_wall_time_us":61475}
I20250626 14:04:34.027724 2282 ts_tablet_manager.cc:1918] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.027851 2282 log.cc:1199] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/6a0b30a82a024246a45dc64dce32947d
I20250626 14:04:34.028163 1658 tablet_service.cc:1515] Processing DeleteTablet for tablet 6a0b30a82a024246a45dc64dce32947d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 14:04:33 UTC) from {username='slave'} at 127.0.0.1:43716
I20250626 14:04:34.028422 1657 tablet_service.cc:1515] Processing DeleteTablet for tablet d0f427caf3744354ae36c40f257ec89c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 14:04:34 UTC) from {username='slave'} at 127.0.0.1:43716
I20250626 14:04:34.028226 2282 ts_tablet_manager.cc:1939] T 6a0b30a82a024246a45dc64dce32947d P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:34.028789 2283 tablet.cc:1722] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 14:04:34.029247 1330 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet 6a0b30a82a024246a45dc64dce32947d (table default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81]) successfully deleted
I20250626 14:04:34.029356 2284 tablet.cc:1722] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 14:04:34.029523 2282 tablet_replica.cc:331] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:34.029628 2282 raft_consensus.cc:2241] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:34.029721 2282 raft_consensus.cc:2270] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:34.030282 2282 ts_tablet_manager.cc:1905] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:34.032127 2282 ts_tablet_manager.cc:1918] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.032359 2282 log.cc:1199] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/d0f427caf3744354ae36c40f257ec89c
I20250626 14:04:34.032706 2282 ts_tablet_manager.cc:1939] T d0f427caf3744354ae36c40f257ec89c P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:34.033398 1330 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet d0f427caf3744354ae36c40f257ec89c (table default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738]) successfully deleted
I20250626 14:04:34.033790 2285 tablet_replica.cc:331] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:34.033988 2285 raft_consensus.cc:2241] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:34.034180 2285 raft_consensus.cc:2270] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:34.035051 2285 ts_tablet_manager.cc:1905] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:34.037318 2285 ts_tablet_manager.cc:1918] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.037443 2285 log.cc:1199] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/6a0b30a82a024246a45dc64dce32947d
I20250626 14:04:34.037786 2285 ts_tablet_manager.cc:1939] T 6a0b30a82a024246a45dc64dce32947d P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:34.038506 1331 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet 6a0b30a82a024246a45dc64dce32947d (table default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81]) successfully deleted
I20250626 14:04:34.038776 2285 tablet_replica.cc:331] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:34.038942 2285 raft_consensus.cc:2241] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:34.039100 2285 raft_consensus.cc:2270] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:34.039278 1792 tablet_service.cc:1515] Processing DeleteTablet for tablet 6a0b30a82a024246a45dc64dce32947d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 14:04:33 UTC) from {username='slave'} at 127.0.0.1:48176
I20250626 14:04:34.039278 1793 tablet_service.cc:1515] Processing DeleteTablet for tablet d0f427caf3744354ae36c40f257ec89c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 14:04:34 UTC) from {username='slave'} at 127.0.0.1:48176
I20250626 14:04:34.039913 2285 ts_tablet_manager.cc:1905] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:34.040047 2286 tablet.cc:1722] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250626 14:04:34.041977 2285 ts_tablet_manager.cc:1918] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.042099 2285 log.cc:1199] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/d0f427caf3744354ae36c40f257ec89c
I20250626 14:04:34.043726 2285 ts_tablet_manager.cc:1939] T d0f427caf3744354ae36c40f257ec89c P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:34.044358 2287 tablet_replica.cc:331] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:34.044710 2287 raft_consensus.cc:2241] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:34.045267 2287 raft_consensus.cc:2270] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:34.045367 1331 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet d0f427caf3744354ae36c40f257ec89c (table default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738]) successfully deleted
I20250626 14:04:34.045976 2287 ts_tablet_manager.cc:1905] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:34.047715 2287 ts_tablet_manager.cc:1918] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.047845 2287 log.cc:1199] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/6a0b30a82a024246a45dc64dce32947d
I20250626 14:04:34.048162 2287 ts_tablet_manager.cc:1939] T 6a0b30a82a024246a45dc64dce32947d P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:34.048991 1330 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet 6a0b30a82a024246a45dc64dce32947d (table default.table_68167203433a4b67aadc617167cce998 [id=fa7ec8992cb04805a27ff28512572a81]) successfully deleted
I20250626 14:04:34.049247 2287 tablet_replica.cc:331] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:34.049403 2287 raft_consensus.cc:2241] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:34.049646 2287 raft_consensus.cc:2270] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:34.050211 2287 ts_tablet_manager.cc:1905] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:34.051962 2287 ts_tablet_manager.cc:1918] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.052030 2287 log.cc:1199] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/d0f427caf3744354ae36c40f257ec89c
I20250626 14:04:34.052258 2287 ts_tablet_manager.cc:1939] T d0f427caf3744354ae36c40f257ec89c P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:34.052973 1330 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet d0f427caf3744354ae36c40f257ec89c (table default.table_8a07386d09424c52995940144f8d5d3c [id=4f2719251ee84bc9af2e04e05aa3b738]) successfully deleted
I20250626 14:04:34.109546 1478 catalog_manager.cc:2149] created HMS catalog entry for table [id=6784347b414a45f785c279bf0ab4800e]
W20250626 14:04:34.110169 1478 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 81db61eed5584e7e8b0cf40e5441f88e is not leader of this config: current role FOLLOWER
I20250626 14:04:34.179308 1561 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 14:04:34.194694 1478 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=6784347b414a45f785c279bf0ab4800e]
I20250626 14:04:34.194888 1342 catalog_manager.cc:3460] renamed table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75] in HMS: new name default.table_7d5a9b0c3ae440b89dac3ca5dd8e81ef
I20250626 14:04:34.195089 1478 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51198 (request call id 30) took 1286 ms. Trace:
I20250626 14:04:34.195165 1478 rpcz_store.cc:276] 0626 14:04:32.908555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 14:04:33.172230 (+263675us) service_pool.cc:225] Handling call
0626 14:04:33.971462 (+799232us) catalog_manager.cc:2076] Acquired catalog manager lock
0626 14:04:33.971834 (+ 372us) catalog_manager.cc:2122] Created new table and tablet info
0626 14:04:34.109536 (+137702us) catalog_manager.cc:2148] Created new table in HMS catalog
0626 14:04:34.110127 (+ 591us) write_op.cc:489] Releasing partition, row and schema locks
0626 14:04:34.110129 (+ 2us) write_op.cc:454] Released schema lock
0626 14:04:34.110251 (+ 122us) catalog_manager.cc:2156] Rolling back HMS table creation
0626 14:04:34.195069 (+ 84818us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 14:04:34.110077 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 81db61eed5584e7e8b0cf40e5441f88e is not leader of this config: current role FOLLOWER)
0626 14:04:34.110085 (+ 8us) write_op.cc:489] Releasing partition, row and schema locks
0626 14:04:34.110090 (+ 5us) write_op.cc:454] Released schema lock
0626 14:04:34.110093 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":52442,"Hive Metastore.run_cpu_time_us":427,"Hive Metastore.run_wall_time_us":169394,"child_traces":[["op",{}]]}
I20250626 14:04:34.253728 1344 catalog_manager.cc:3460] renamed table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447] in HMS: new name default.table_f01c05221e5c4e75bf700b9474072881
I20250626 14:04:34.279495 1336 catalog_manager.cc:2149] created HMS catalog entry for table [id=61127b0a117a4fbaa8bbbbb397cbc6bb]
F20250626 14:04:34.284759 1542 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 16, previous 16; dumping first 4 out of 6 received events: NotificationEvent(eventId=14, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_72f9307d2f5247a0ae4499c953ea1650, message={"server":"","servicePrincipal":"","db":"default","table":"table_72f9307d2f5247a0ae4499c953ea1650","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_72f9307d2f5247a0ae4499c953ea1650\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_72f9307d2f5247a0ae4499c953ea1650\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_72f9307d2f5247a0ae4499c953ea1650\",\"kudu.table_id\":\"61be84e63b2e478b9da5619fdbdffee9\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=15, eventTime=1750946674, eventType=CREATE_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=ALTER_TABLE, dbName=default, tableName=table_7d5a9b0c3ae440b89dac3ca5dd8e81ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_fb8ebc7aa8894f28b4d79252382f356a","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_fb8ebc7aa8894f28b4d79252382f356a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_fb8ebc7aa8894f28b4d79252382f356a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fb8ebc7aa8894f28b4d79252382f356a\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250626 14:04:34.291294 1657 tablet_service.cc:1468] Processing CreateTablet for tablet b325108f7de949199cc10dbe259a0651 (DEFAULT_TABLE table=default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.291484 1793 tablet_service.cc:1468] Processing CreateTablet for tablet b325108f7de949199cc10dbe259a0651 (DEFAULT_TABLE table=default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.291625 1657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b325108f7de949199cc10dbe259a0651. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:34.291733 1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b325108f7de949199cc10dbe259a0651. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:34.294458 1928 tablet_service.cc:1468] Processing CreateTablet for tablet b325108f7de949199cc10dbe259a0651 (DEFAULT_TABLE table=default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.294770 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b325108f7de949199cc10dbe259a0651. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:34.298463 2290 tablet_bootstrap.cc:492] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:34.299752 2290 tablet_bootstrap.cc:654] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:34.299963 1336 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233] alter complete (version 1)
I20250626 14:04:34.300140 2289 tablet_bootstrap.cc:492] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:34.301972 2289 tablet_bootstrap.cc:654] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:34.303509 2290 tablet_bootstrap.cc:492] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:34.303601 2290 ts_tablet_manager.cc:1397] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250626 14:04:34.304126 2290 raft_consensus.cc:357] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:34.304250 2290 raft_consensus.cc:383] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.304291 2290 raft_consensus.cc:738] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.304428 2290 consensus_queue.cc:260] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:34.304723 2290 ts_tablet_manager.cc:1428] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:34.305367 1336 master_service.cc:948] ReplaceTablet: received request to replace tablet ace98628290843938dc5db70e5a43a37 from {username='slave'} at 127.0.0.1:59552
I20250626 14:04:34.311878 1336 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Sending DeleteTablet for 3 replicas of tablet ace98628290843938dc5db70e5a43a37
I20250626 14:04:34.312443 1657 tablet_service.cc:1515] Processing DeleteTablet for tablet ace98628290843938dc5db70e5a43a37 with delete_type TABLET_DATA_DELETED (replaced by tablet 0db0b418e8b8412da3dc098ef6aab940) from {username='slave'} at 127.0.0.1:43716
I20250626 14:04:34.312825 1793 tablet_service.cc:1515] Processing DeleteTablet for tablet ace98628290843938dc5db70e5a43a37 with delete_type TABLET_DATA_DELETED (replaced by tablet 0db0b418e8b8412da3dc098ef6aab940) from {username='slave'} at 127.0.0.1:48176
I20250626 14:04:34.312851 1928 tablet_service.cc:1515] Processing DeleteTablet for tablet ace98628290843938dc5db70e5a43a37 with delete_type TABLET_DATA_DELETED (replaced by tablet 0db0b418e8b8412da3dc098ef6aab940) from {username='slave'} at 127.0.0.1:54652
I20250626 14:04:34.313017 2287 tablet_replica.cc:331] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:34.313129 2285 tablet_replica.cc:331] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:34.313172 2287 raft_consensus.cc:2241] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:34.313244 2285 raft_consensus.cc:2241] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:34.313289 2287 raft_consensus.cc:2270] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:34.313494 2285 raft_consensus.cc:2270] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:34.313544 2282 tablet_replica.cc:331] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:34.313683 2282 raft_consensus.cc:2241] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:34.313796 2282 raft_consensus.cc:2270] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:34.313946 1336 catalog_manager.cc:6652] ReplaceTablet: tablet ace98628290843938dc5db70e5a43a37 deleted and replaced by tablet 0db0b418e8b8412da3dc098ef6aab940
I20250626 14:04:34.313957 2287 ts_tablet_manager.cc:1905] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:34.314054 2285 ts_tablet_manager.cc:1905] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:34.315321 1336 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59552:
name: "default.Table_f7012cd4bf5d40d6af93cf9b01405dcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:34.315727 2291 tablet_bootstrap.cc:492] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:34.316243 2285 ts_tablet_manager.cc:1918] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.316244 2287 ts_tablet_manager.cc:1918] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.316311 2285 log.cc:1199] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/ace98628290843938dc5db70e5a43a37
I20250626 14:04:34.316313 2287 log.cc:1199] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/ace98628290843938dc5db70e5a43a37
I20250626 14:04:34.316954 2291 tablet_bootstrap.cc:654] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:34.318188 2289 tablet_bootstrap.cc:492] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:34.318279 2289 ts_tablet_manager.cc:1397] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.002s
I20250626 14:04:34.318902 2289 raft_consensus.cc:357] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:34.319043 2289 raft_consensus.cc:383] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.319080 2289 raft_consensus.cc:738] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.319254 2289 consensus_queue.cc:260] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:34.319602 2289 ts_tablet_manager.cc:1428] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:34.320112 2287 ts_tablet_manager.cc:1939] T ace98628290843938dc5db70e5a43a37 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:34.320454 2282 ts_tablet_manager.cc:1905] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:34.320853 2291 tablet_bootstrap.cc:492] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:34.322059 2291 ts_tablet_manager.cc:1397] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250626 14:04:34.322455 1928 tablet_service.cc:1468] Processing CreateTablet for tablet 0db0b418e8b8412da3dc098ef6aab940 (DEFAULT_TABLE table=default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.322572 2291 raft_consensus.cc:357] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:34.322719 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db0b418e8b8412da3dc098ef6aab940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:34.322682 2291 raft_consensus.cc:383] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.322779 2291 raft_consensus.cc:738] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.322924 2291 consensus_queue.cc:260] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
@ 0x7f888db9adcd google::LogMessage::Fail() at ??:0
I20250626 14:04:34.323519 2291 ts_tablet_manager.cc:1428] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:34.324379 1793 tablet_service.cc:1468] Processing CreateTablet for tablet 0db0b418e8b8412da3dc098ef6aab940 (DEFAULT_TABLE table=default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.326299 1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db0b418e8b8412da3dc098ef6aab940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:34.327979 1343 catalog_manager.cc:2149] created HMS catalog entry for table [id=cdd0994532564911854d5ce33557b62c]
@ 0x7f888db9eb93 google::LogMessage::SendToLog() at ??:0
I20250626 14:04:34.333962 1343 master_service.cc:948] ReplaceTablet: received request to replace tablet 071dcb923e314fca9fbbf94592b01fe2 from {username='slave'} at 127.0.0.1:59552
I20250626 14:04:34.334360 1658 tablet_service.cc:1468] Processing CreateTablet for tablet 0db0b418e8b8412da3dc098ef6aab940 (DEFAULT_TABLE table=default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.334652 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db0b418e8b8412da3dc098ef6aab940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:34.342679 2289 tablet_bootstrap.cc:492] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:34.343259 2290 tablet_bootstrap.cc:492] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:34.343806 2289 tablet_bootstrap.cc:654] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:34.344699 1793 tablet_service.cc:1468] Processing CreateTablet for tablet 8e965ba7276e469c94493334ba18fa1b (DEFAULT_TABLE table=default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.344933 1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e965ba7276e469c94493334ba18fa1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:34.346144 1658 tablet_service.cc:1468] Processing CreateTablet for tablet 8e965ba7276e469c94493334ba18fa1b (DEFAULT_TABLE table=default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.346410 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e965ba7276e469c94493334ba18fa1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:34.347055 1927 tablet_service.cc:1468] Processing CreateTablet for tablet 8e965ba7276e469c94493334ba18fa1b (DEFAULT_TABLE table=default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.322582 2282 ts_tablet_manager.cc:1918] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.347214 2282 log.cc:1199] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/ace98628290843938dc5db70e5a43a37
I20250626 14:04:34.347430 2282 ts_tablet_manager.cc:1939] T ace98628290843938dc5db70e5a43a37 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:34.347807 2285 ts_tablet_manager.cc:1939] T ace98628290843938dc5db70e5a43a37 P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:34.348014 1343 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Sending DeleteTablet for 3 replicas of tablet 071dcb923e314fca9fbbf94592b01fe2
I20250626 14:04:34.348276 1343 catalog_manager.cc:6652] ReplaceTablet: tablet 071dcb923e314fca9fbbf94592b01fe2 deleted and replaced by tablet d21f45117538444ba7b968411a58f099
I20250626 14:04:34.348507 2290 tablet_bootstrap.cc:654] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:34.349145 1926 tablet_service.cc:1515] Processing DeleteTablet for tablet 071dcb923e314fca9fbbf94592b01fe2 with delete_type TABLET_DATA_DELETED (replaced by tablet d21f45117538444ba7b968411a58f099) from {username='slave'} at 127.0.0.1:54652
I20250626 14:04:34.349463 1343 master_service.cc:948] ReplaceTablet: received request to replace tablet ace98628290843938dc5db70e5a43a37 from {username='slave'} at 127.0.0.1:59552
I20250626 14:04:34.349746 1330 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet ace98628290843938dc5db70e5a43a37 (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]) successfully deleted
I20250626 14:04:34.349771 1343 master_service.cc:948] ReplaceTablet: received request to replace tablet d2a95d2f060943a68721021a204e0cc3 from {username='slave'} at 127.0.0.1:59552
I20250626 14:04:34.350639 1927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e965ba7276e469c94493334ba18fa1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:34.353892 2291 tablet_bootstrap.cc:492] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:34.354839 1793 tablet_service.cc:1515] Processing DeleteTablet for tablet 071dcb923e314fca9fbbf94592b01fe2 with delete_type TABLET_DATA_DELETED (replaced by tablet d21f45117538444ba7b968411a58f099) from {username='slave'} at 127.0.0.1:48176
I20250626 14:04:34.355006 2287 tablet_replica.cc:331] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:34.355130 2287 raft_consensus.cc:2241] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:34.355232 2287 raft_consensus.cc:2270] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:34.356429 2291 tablet_bootstrap.cc:654] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:34.356820 1331 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet ace98628290843938dc5db70e5a43a37 (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]) successfully deleted
I20250626 14:04:34.358021 2282 tablet_replica.cc:331] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:34.358136 2282 raft_consensus.cc:2241] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:34.358189 1330 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet ace98628290843938dc5db70e5a43a37 (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]) successfully deleted
I20250626 14:04:34.358316 2282 raft_consensus.cc:2270] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:34.359030 2282 ts_tablet_manager.cc:1905] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f888db9a7cc google::LogMessage::Flush() at ??:0
I20250626 14:04:34.360911 2291 tablet_bootstrap.cc:492] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:34.360992 2291 ts_tablet_manager.cc:1397] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250626 14:04:34.361155 2282 ts_tablet_manager.cc:1918] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.361215 2282 log.cc:1199] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/071dcb923e314fca9fbbf94592b01fe2
@ 0x7f888db9aa11 google::LogMessage::~LogMessage() at ??:0
I20250626 14:04:34.361465 2282 ts_tablet_manager.cc:1939] T 071dcb923e314fca9fbbf94592b01fe2 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:34.361542 2291 raft_consensus.cc:357] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.361654 2291 raft_consensus.cc:383] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.361703 2291 raft_consensus.cc:738] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.361869 2291 consensus_queue.cc:260] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.362035 1330 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet 071dcb923e314fca9fbbf94592b01fe2 (table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]) successfully deleted
I20250626 14:04:34.362200 2291 ts_tablet_manager.cc:1428] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:34.362382 2291 tablet_bootstrap.cc:492] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:34.363637 2291 tablet_bootstrap.cc:654] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
@ 0x7f8892b33c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250626 14:04:34.366309 1657 tablet_service.cc:1515] Processing DeleteTablet for tablet 071dcb923e314fca9fbbf94592b01fe2 with delete_type TABLET_DATA_DELETED (replaced by tablet d21f45117538444ba7b968411a58f099) from {username='slave'} at 127.0.0.1:43716
I20250626 14:04:34.366596 2285 tablet_replica.cc:331] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:34.366724 2285 raft_consensus.cc:2241] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:34.366832 2285 raft_consensus.cc:2270] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:34.367473 1343 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Sending DeleteTablet for 3 replicas of tablet d2a95d2f060943a68721021a204e0cc3
I20250626 14:04:34.368269 1928 tablet_service.cc:1515] Processing DeleteTablet for tablet d2a95d2f060943a68721021a204e0cc3 with delete_type TABLET_DATA_DELETED (replaced by tablet a4e9c6ce944b4ec6af44f7852cac93d3) from {username='slave'} at 127.0.0.1:54652
I20250626 14:04:34.368438 2282 tablet_replica.cc:331] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:34.368582 2282 raft_consensus.cc:2241] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:34.368605 1343 catalog_manager.cc:6652] ReplaceTablet: tablet d2a95d2f060943a68721021a204e0cc3 deleted and replaced by tablet a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:04:34.368669 1793 tablet_service.cc:1515] Processing DeleteTablet for tablet d2a95d2f060943a68721021a204e0cc3 with delete_type TABLET_DATA_DELETED (replaced by tablet a4e9c6ce944b4ec6af44f7852cac93d3) from {username='slave'} at 127.0.0.1:48176
I20250626 14:04:34.368721 2282 raft_consensus.cc:2270] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:34.370108 1343 master_service.cc:948] ReplaceTablet: received request to replace tablet 6a0b30a82a024246a45dc64dce32947d from {username='slave'} at 127.0.0.1:59552
I20250626 14:04:34.370958 1928 tablet_service.cc:1515] Processing DeleteTablet for tablet 071dcb923e314fca9fbbf94592b01fe2 with delete_type TABLET_DATA_DELETED (replaced by tablet d21f45117538444ba7b968411a58f099) from {username='slave'} at 127.0.0.1:54652
I20250626 14:04:34.371395 2282 ts_tablet_manager.cc:1905] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:34.373575 1928 tablet_service.cc:1468] Processing CreateTablet for tablet d21f45117538444ba7b968411a58f099 (DEFAULT_TABLE table=default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.373576 1793 tablet_service.cc:1468] Processing CreateTablet for tablet d21f45117538444ba7b968411a58f099 (DEFAULT_TABLE table=default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.373814 1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d21f45117538444ba7b968411a58f099. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:34.373814 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d21f45117538444ba7b968411a58f099. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:34.376114 2282 ts_tablet_manager.cc:1918] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.376185 2282 log.cc:1199] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/d2a95d2f060943a68721021a204e0cc3
I20250626 14:04:34.376489 2282 ts_tablet_manager.cc:1939] T d2a95d2f060943a68721021a204e0cc3 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:34.377358 1330 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet d2a95d2f060943a68721021a204e0cc3 (table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]) successfully deleted
W20250626 14:04:34.377652 1330 catalog_manager.cc:4895] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): delete failed for tablet 071dcb923e314fca9fbbf94592b01fe2 because the tablet was not found. No further retry: Not found: Tablet not found: 071dcb923e314fca9fbbf94592b01fe2
I20250626 14:04:34.378211 1657 tablet_service.cc:1515] Processing DeleteTablet for tablet d2a95d2f060943a68721021a204e0cc3 with delete_type TABLET_DATA_DELETED (replaced by tablet a4e9c6ce944b4ec6af44f7852cac93d3) from {username='slave'} at 127.0.0.1:43716
I20250626 14:04:34.378377 1656 tablet_service.cc:1468] Processing CreateTablet for tablet d21f45117538444ba7b968411a58f099 (DEFAULT_TABLE table=default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.378656 1656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d21f45117538444ba7b968411a58f099. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:34.380018 1792 tablet_service.cc:1468] Processing CreateTablet for tablet a4e9c6ce944b4ec6af44f7852cac93d3 (DEFAULT_TABLE table=default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.380023 1928 tablet_service.cc:1468] Processing CreateTablet for tablet a4e9c6ce944b4ec6af44f7852cac93d3 (DEFAULT_TABLE table=default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.380219 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e9c6ce944b4ec6af44f7852cac93d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:34.380234 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e9c6ce944b4ec6af44f7852cac93d3. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f8892b32ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250626 14:04:34.382556 1657 tablet_service.cc:1468] Processing CreateTablet for tablet a4e9c6ce944b4ec6af44f7852cac93d3 (DEFAULT_TABLE table=default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:34.382771 1657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e9c6ce944b4ec6af44f7852cac93d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:34.385272 2285 ts_tablet_manager.cc:1905] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f8892b3230b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250626 14:04:34.387324 2285 ts_tablet_manager.cc:1918] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.387393 2285 log.cc:1199] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/071dcb923e314fca9fbbf94592b01fe2
I20250626 14:04:34.387665 2285 ts_tablet_manager.cc:1939] T 071dcb923e314fca9fbbf94592b01fe2 P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:34.388382 1331 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet 071dcb923e314fca9fbbf94592b01fe2 (table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]) successfully deleted
I20250626 14:04:34.388968 2287 ts_tablet_manager.cc:1905] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f8892b37a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56478d330752 std::function<>::operator()() at ??:0
I20250626 14:04:34.391222 2287 ts_tablet_manager.cc:1918] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.391290 2287 log.cc:1199] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/071dcb923e314fca9fbbf94592b01fe2
I20250626 14:04:34.391556 2287 ts_tablet_manager.cc:1939] T 071dcb923e314fca9fbbf94592b01fe2 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:34.392254 1330 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet 071dcb923e314fca9fbbf94592b01fe2 (table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]) successfully deleted
@ 0x7f888efec0a1 kudu::Thread::SuperviseThread() at ??:0
I20250626 14:04:34.393183 1341 catalog_manager.cc:2149] created HMS catalog entry for table [id=2d38a93c3aca41cfb07d20ca6241d843]
@ 0x7f888f7db6db start_thread at ??:0
I20250626 14:04:34.396196 1343 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59552:
table { table_name: "default.Table_afd26acd01c7422fbb06520af5a4f76c" } new_table_name: "default.Table_4d81e86bd6614c3db77efeaca0daa56f" modify_external_catalogs: true
@ 0x7f888cd5171f clone at ??:0
I20250626 14:04:34.398619 2287 tablet_replica.cc:331] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:34.398732 2287 raft_consensus.cc:2241] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:34.398885 2287 raft_consensus.cc:2270] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:34.399047 2285 tablet_replica.cc:331] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:34.399164 2285 raft_consensus.cc:2241] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:34.399260 2285 raft_consensus.cc:2270] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:34.399380 2287 ts_tablet_manager.cc:1905] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:34.399829 2285 ts_tablet_manager.cc:1905] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:34.400280 2290 tablet_bootstrap.cc:492] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:34.400354 2290 ts_tablet_manager.cc:1397] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.000s
I20250626 14:04:34.400395 2289 tablet_bootstrap.cc:492] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:34.400490 2289 ts_tablet_manager.cc:1397] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.058s user 0.000s sys 0.002s
I20250626 14:04:34.400992 2289 raft_consensus.cc:357] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.400978 2290 raft_consensus.cc:357] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.401087 2289 raft_consensus.cc:383] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.401105 2290 raft_consensus.cc:383] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.401115 2289 raft_consensus.cc:738] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.401144 2290 raft_consensus.cc:738] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.401238 2289 consensus_queue.cc:260] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.401311 2290 consensus_queue.cc:260] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.401574 2289 ts_tablet_manager.cc:1428] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:34.401655 2290 ts_tablet_manager.cc:1428] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 14:04:34.401675 2287 ts_tablet_manager.cc:1918] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.401741 2287 log.cc:1199] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/d2a95d2f060943a68721021a204e0cc3
I20250626 14:04:34.401772 2289 tablet_bootstrap.cc:492] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:34.401857 2290 tablet_bootstrap.cc:492] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:34.401981 2287 ts_tablet_manager.cc:1939] T d2a95d2f060943a68721021a204e0cc3 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:34.402766 2289 tablet_bootstrap.cc:654] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:34.402945 2290 tablet_bootstrap.cc:654] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:34.403821 2289 tablet_bootstrap.cc:492] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:34.403891 2289 ts_tablet_manager.cc:1397] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:34.404299 2289 raft_consensus.cc:357] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.404394 2289 raft_consensus.cc:383] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.404425 2289 raft_consensus.cc:738] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.404574 2289 consensus_queue.cc:260] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.404887 2289 ts_tablet_manager.cc:1428] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:34.405184 2289 tablet_bootstrap.cc:492] T d21f45117538444ba7b968411a58f099 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:34.406167 2289 tablet_bootstrap.cc:654] T d21f45117538444ba7b968411a58f099 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:34.407162 2289 tablet_bootstrap.cc:492] T d21f45117538444ba7b968411a58f099 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:34.407231 2289 ts_tablet_manager.cc:1397] T d21f45117538444ba7b968411a58f099 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:34.407668 2289 raft_consensus.cc:357] T d21f45117538444ba7b968411a58f099 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.407773 2289 raft_consensus.cc:383] T d21f45117538444ba7b968411a58f099 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.407804 2289 raft_consensus.cc:738] T d21f45117538444ba7b968411a58f099 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.407917 2289 consensus_queue.cc:260] T d21f45117538444ba7b968411a58f099 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.408077 2285 ts_tablet_manager.cc:1918] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:34.408145 2285 log.cc:1199] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/d2a95d2f060943a68721021a204e0cc3
I20250626 14:04:34.408210 2289 ts_tablet_manager.cc:1428] T d21f45117538444ba7b968411a58f099 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:34.408383 2289 tablet_bootstrap.cc:492] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:34.408387 2285 ts_tablet_manager.cc:1939] T d2a95d2f060943a68721021a204e0cc3 P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:34.409351 2289 tablet_bootstrap.cc:654] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:34.411396 2290 tablet_bootstrap.cc:492] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:34.411489 2290 ts_tablet_manager.cc:1397] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250626 14:04:34.411907 2290 raft_consensus.cc:357] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.411984 2290 raft_consensus.cc:383] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.412005 2290 raft_consensus.cc:738] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.412110 2290 consensus_queue.cc:260] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.412858 2290 ts_tablet_manager.cc:1428] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:34.413028 2290 tablet_bootstrap.cc:492] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:34.414134 2290 tablet_bootstrap.cc:654] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:34.417726 2289 tablet_bootstrap.cc:492] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:34.417744 2291 tablet_bootstrap.cc:492] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:34.417829 2289 ts_tablet_manager.cc:1397] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250626 14:04:34.417860 2291 ts_tablet_manager.cc:1397] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.000s
I20250626 14:04:34.418234 2289 raft_consensus.cc:357] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.418314 2289 raft_consensus.cc:383] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.418335 2289 raft_consensus.cc:738] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.418334 2291 raft_consensus.cc:357] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.418433 2291 raft_consensus.cc:383] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.418465 2291 raft_consensus.cc:738] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.418447 2289 consensus_queue.cc:260] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.418592 2291 consensus_queue.cc:260] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.418720 2289 ts_tablet_manager.cc:1428] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 14:04:34.418864 2291 ts_tablet_manager.cc:1428] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:34.419029 2291 tablet_bootstrap.cc:492] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:34.420033 2291 tablet_bootstrap.cc:654] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:34.427378 2291 tablet_bootstrap.cc:492] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:34.427476 2291 ts_tablet_manager.cc:1397] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250626 14:04:34.428143 2291 raft_consensus.cc:357] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.428267 2291 raft_consensus.cc:383] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.428295 2291 raft_consensus.cc:738] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.428459 2291 consensus_queue.cc:260] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.429343 2291 ts_tablet_manager.cc:1428] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:34.429543 2291 tablet_bootstrap.cc:492] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:34.430681 2290 tablet_bootstrap.cc:492] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:34.430784 2290 ts_tablet_manager.cc:1397] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20250626 14:04:34.430837 2291 tablet_bootstrap.cc:654] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:34.431277 2290 raft_consensus.cc:357] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.431380 2290 raft_consensus.cc:383] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.431412 2290 raft_consensus.cc:738] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.431576 2290 consensus_queue.cc:260] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.431928 2290 ts_tablet_manager.cc:1428] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:34.434212 2290 tablet_bootstrap.cc:492] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:34.435446 2290 tablet_bootstrap.cc:654] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:34.435856 2291 tablet_bootstrap.cc:492] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:34.435925 2291 ts_tablet_manager.cc:1397] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250626 14:04:34.436483 2291 raft_consensus.cc:357] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.436614 2291 raft_consensus.cc:383] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.436627 2290 tablet_bootstrap.cc:492] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:34.436653 2291 raft_consensus.cc:738] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.436702 2290 ts_tablet_manager.cc:1397] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:34.436789 2291 consensus_queue.cc:260] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.437048 2291 ts_tablet_manager.cc:1428] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:34.437161 2290 raft_consensus.cc:357] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.437260 2290 raft_consensus.cc:383] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:34.437292 2290 raft_consensus.cc:738] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:34.437400 2290 consensus_queue.cc:260] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.438880 2290 ts_tablet_manager.cc:1428] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 14:04:34.485857 2288 raft_consensus.cc:491] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:34.486011 2288 raft_consensus.cc:513] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.486459 2288 leader_election.cc:290] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:34.486975 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e965ba7276e469c94493334ba18fa1b" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:34.486981 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e965ba7276e469c94493334ba18fa1b" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:34.487130 1813 raft_consensus.cc:2466] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 0.
I20250626 14:04:34.487130 1947 raft_consensus.cc:2466] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 0.
I20250626 14:04:34.487547 1613 leader_election.cc:304] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:34.487782 2288 raft_consensus.cc:2802] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:34.487844 2288 raft_consensus.cc:491] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:34.487884 2288 raft_consensus.cc:3058] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.488843 2288 raft_consensus.cc:513] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.489168 2288 leader_election.cc:290] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 election: Requested vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:34.489573 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e965ba7276e469c94493334ba18fa1b" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:34.489603 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e965ba7276e469c94493334ba18fa1b" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:34.489702 1947 raft_consensus.cc:3058] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.489718 1813 raft_consensus.cc:3058] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.490712 1947 raft_consensus.cc:2466] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
I20250626 14:04:34.490717 1813 raft_consensus.cc:2466] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
I20250626 14:04:34.491130 1613 leader_election.cc:304] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:34.491369 2288 raft_consensus.cc:2802] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:34.491470 2288 raft_consensus.cc:695] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [term 1 LEADER]: Becoming Leader. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Running, Role: LEADER
I20250626 14:04:34.491616 2288 consensus_queue.cc:237] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
W20250626 14:04:34.601027 1611 connection.cc:537] server connection from 127.0.0.1:43716 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:34.601054 1748 connection.cc:537] server connection from 127.0.0.1:48176 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:34.601087 2001 connection.cc:537] client connection to 127.1.11.254:36707 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:34.601711 1746 connection.cc:537] client connection to 127.1.11.254:36707 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:34.601750 1611 connection.cc:537] client connection to 127.1.11.254:36707 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:34.602627 1881 connection.cc:537] client connection to 127.1.11.254:36707 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:34.603808 2124 connection.cc:537] server connection from 127.0.0.1:49328 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:34.604950 2322 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 14:04:34.602394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.11.254:36707 (local address 127.1.11.194:46313)
0626 14:04:34.604143 (+ 1749us) negotiation.cc:107] Waiting for socket to connect
0626 14:04:34.604204 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":1635,"thread_start_us":78,"threads_started":1}
W20250626 14:04:34.606079 2321 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 14:04:34.602386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.11.254:36707 (local address 127.1.11.193:35561)
0626 14:04:34.605366 (+ 2980us) negotiation.cc:107] Waiting for socket to connect
0626 14:04:34.605425 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":2877,"thread_start_us":70,"threads_started":1}
I20250626 14:04:34.650185 2069 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:34.650331 2069 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.650763 2069 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:34.651211 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4e9c6ce944b4ec6af44f7852cac93d3" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:34.651353 1677 raft_consensus.cc:2466] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:34.651276 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4e9c6ce944b4ec6af44f7852cac93d3" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:34.651477 1947 raft_consensus.cc:2466] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:34.651669 1749 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:34.651882 2069 raft_consensus.cc:2802] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:34.651932 2069 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:34.651965 2069 raft_consensus.cc:3058] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.653016 2069 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.653368 2069 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:34.653918 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4e9c6ce944b4ec6af44f7852cac93d3" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:34.654031 1947 raft_consensus.cc:3058] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.654961 1947 raft_consensus.cc:2466] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:34.655007 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4e9c6ce944b4ec6af44f7852cac93d3" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:34.655143 1677 raft_consensus.cc:3058] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.655309 1748 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:34.655567 2069 raft_consensus.cc:2802] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:34.655702 2069 raft_consensus.cc:695] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Becoming Leader. State: Replica: e517893e13f546c0b40d976633b49651, State: Running, Role: LEADER
I20250626 14:04:34.655848 2069 consensus_queue.cc:237] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.656394 1677 raft_consensus.cc:2466] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:34.670619 2083 raft_consensus.cc:491] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:34.670751 2083 raft_consensus.cc:513] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:34.671119 2083 leader_election.cc:290] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:34.671509 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b325108f7de949199cc10dbe259a0651" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:34.671687 1677 raft_consensus.cc:2466] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:34.671743 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b325108f7de949199cc10dbe259a0651" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:34.671890 1813 raft_consensus.cc:2466] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:34.672021 1884 leader_election.cc:304] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:34.672335 2083 raft_consensus.cc:2802] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:34.672403 2083 raft_consensus.cc:491] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:34.672442 2083 raft_consensus.cc:3058] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.673790 2083 raft_consensus.cc:513] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:34.674263 2083 leader_election.cc:290] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Requested vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:34.674544 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b325108f7de949199cc10dbe259a0651" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:34.674682 1677 raft_consensus.cc:3058] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.674786 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b325108f7de949199cc10dbe259a0651" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:34.674924 1813 raft_consensus.cc:3058] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.675626 1677 raft_consensus.cc:2466] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:34.675901 1884 leader_election.cc:304] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:34.676062 2083 raft_consensus.cc:2802] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:34.676156 1813 raft_consensus.cc:2466] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:34.676467 2083 raft_consensus.cc:695] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Running, Role: LEADER
I20250626 14:04:34.676669 2083 consensus_queue.cc:237] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:34.677379 2288 raft_consensus.cc:491] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:34.677487 2288 raft_consensus.cc:513] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.677932 2288 leader_election.cc:290] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:34.678649 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d21f45117538444ba7b968411a58f099" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:34.678632 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d21f45117538444ba7b968411a58f099" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:34.678762 1947 raft_consensus.cc:2466] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 0.
I20250626 14:04:34.678762 1813 raft_consensus.cc:2466] T d21f45117538444ba7b968411a58f099 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 0.
I20250626 14:04:34.679024 1613 leader_election.cc:304] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:34.679348 2288 raft_consensus.cc:2802] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:34.679407 2288 raft_consensus.cc:491] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:34.679467 2288 raft_consensus.cc:3058] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.680311 2288 raft_consensus.cc:513] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.680656 2288 leader_election.cc:290] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 election: Requested vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:34.680985 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d21f45117538444ba7b968411a58f099" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:34.681107 1813 raft_consensus.cc:3058] T d21f45117538444ba7b968411a58f099 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.681105 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d21f45117538444ba7b968411a58f099" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:34.681221 1947 raft_consensus.cc:3058] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.682083 1813 raft_consensus.cc:2466] T d21f45117538444ba7b968411a58f099 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
I20250626 14:04:34.682348 1947 raft_consensus.cc:2466] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
I20250626 14:04:34.682369 1613 leader_election.cc:304] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:34.682615 2288 raft_consensus.cc:2802] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:34.682698 2288 raft_consensus.cc:695] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [term 1 LEADER]: Becoming Leader. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Running, Role: LEADER
I20250626 14:04:34.682832 2288 consensus_queue.cc:237] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.708693 2325 raft_consensus.cc:491] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:34.708851 2325 raft_consensus.cc:513] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.709280 2325 leader_election.cc:290] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:34.709745 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db0b418e8b8412da3dc098ef6aab940" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:34.709887 1677 raft_consensus.cc:2466] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:34.709849 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db0b418e8b8412da3dc098ef6aab940" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:34.710013 1947 raft_consensus.cc:2466] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:34.710182 1749 leader_election.cc:304] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:34.710366 2325 raft_consensus.cc:2802] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:34.710419 2325 raft_consensus.cc:491] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:34.710477 2325 raft_consensus.cc:3058] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.711261 2325 raft_consensus.cc:513] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.711601 2325 leader_election.cc:290] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:34.711961 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db0b418e8b8412da3dc098ef6aab940" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:34.711980 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db0b418e8b8412da3dc098ef6aab940" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:34.712123 1677 raft_consensus.cc:3058] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.712138 1947 raft_consensus.cc:3058] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:34.713165 1677 raft_consensus.cc:2466] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:34.713411 1947 raft_consensus.cc:2466] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:34.713467 1749 leader_election.cc:304] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:34.714191 2325 raft_consensus.cc:2802] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:34.714282 2325 raft_consensus.cc:695] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Becoming Leader. State: Replica: e517893e13f546c0b40d976633b49651, State: Running, Role: LEADER
I20250626 14:04:34.714860 2325 consensus_queue.cc:237] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:34.896127 2330 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 24ad911d71b6433f8c7939a1d1f7b782)
I20250626 14:04:34.896256 2330 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:34.896808 2330 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
I20250626 14:04:34.897298 2149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81db61eed5584e7e8b0cf40e5441f88e" candidate_term: 3 candidate_status { last_received { term: 2 index: 57 } } ignore_live_leader: false dest_uuid: "07532b7648e4414789ed2b099bad2b19" is_pre_election: true
I20250626 14:04:34.897460 2149 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81db61eed5584e7e8b0cf40e5441f88e in term 2.
I20250626 14:04:34.897784 1472 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e; no voters:
I20250626 14:04:34.897965 2330 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 14:04:34.898032 2330 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 24ad911d71b6433f8c7939a1d1f7b782)
I20250626 14:04:34.898082 2330 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 2 FOLLOWER]: Advancing to term 3
W20250626 14:04:34.898445 1470 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
I20250626 14:04:34.899319 2330 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:34.899659 2330 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 3 election: Requested vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
I20250626 14:04:34.899988 2149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81db61eed5584e7e8b0cf40e5441f88e" candidate_term: 3 candidate_status { last_received { term: 2 index: 57 } } ignore_live_leader: false dest_uuid: "07532b7648e4414789ed2b099bad2b19"
I20250626 14:04:34.900143 2149 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 2 FOLLOWER]: Advancing to term 3
W20250626 14:04:34.900468 1470 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
I20250626 14:04:34.901149 2149 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81db61eed5584e7e8b0cf40e5441f88e in term 3.
I20250626 14:04:34.901471 1472 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e; no voters: 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:34.901654 2330 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 3 FOLLOWER]: Leader election won for term 3
I20250626 14:04:34.901827 2330 raft_consensus.cc:695] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 3 LEADER]: Becoming Leader. State: Replica: 81db61eed5584e7e8b0cf40e5441f88e, State: Running, Role: LEADER
I20250626 14:04:34.901997 2330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 2.57, Last appended by leader: 57, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:34.902586 2333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81db61eed5584e7e8b0cf40e5441f88e. Latest consensus state: current_term: 3 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:34.902721 2333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:34.907616 2066 consensus_queue.cc:1035] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:34.912381 2288 consensus_queue.cc:1035] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:35.058121 2325 consensus_queue.cc:1035] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:35.062481 2069 consensus_queue.cc:1035] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:35.118666 2069 consensus_queue.cc:1035] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:35.122634 2069 consensus_queue.cc:1035] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:35.203368 2288 consensus_queue.cc:1035] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:35.206347 2066 consensus_queue.cc:1035] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:35.252830 2083 consensus_queue.cc:1035] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:35.257031 2348 consensus_queue.cc:1035] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 14:04:35.419777 1470 consensus_peers.cc:487] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e -> Peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Couldn't send request to peer 24ad911d71b6433f8c7939a1d1f7b782. Status: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 14:04:35.419778 2354 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:35.419895 2354 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:35.420295 2354 leader_election.cc:290] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:04:35.420910 2124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
I20250626 14:04:35.420938 1496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07532b7648e4414789ed2b099bad2b19" candidate_term: 4 candidate_status { last_received { term: 2 index: 57 } } ignore_live_leader: false dest_uuid: "81db61eed5584e7e8b0cf40e5441f88e" is_pre_election: true
W20250626 14:04:35.421767 2124 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
I20250626 14:04:35.421859 2124 leader_election.cc:304] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19; no voters: 24ad911d71b6433f8c7939a1d1f7b782, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:04:35.422013 2354 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 14:04:35.493224 2149 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 3 FOLLOWER]: Refusing update from remote peer 81db61eed5584e7e8b0cf40e5441f88e: Log matching property violated. Preceding OpId in replica: term: 2 index: 57. Preceding OpId from leader: term: 3 index: 58. (index mismatch)
I20250626 14:04:35.493752 2351 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [LEADER]: Connected to new peer: Peer: permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 56, Time since last communication: 0.000s
I20250626 14:04:35.494855 2354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81db61eed5584e7e8b0cf40e5441f88e. Latest consensus state: current_term: 3 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:35.495007 2354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:35.496315 2351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:35.496434 2351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:35.496496 2354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:35.496618 2354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:35.548343 2334 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 14:04:35.549934 2334 catalog_manager.cc:671] Loaded metadata for table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]
I20250626 14:04:35.550172 2334 catalog_manager.cc:671] Loaded metadata for table default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843]
I20250626 14:04:35.550329 2334 catalog_manager.cc:671] Loaded metadata for table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]
I20250626 14:04:35.550583 2334 catalog_manager.cc:671] Loaded metadata for table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]
I20250626 14:04:35.550736 2334 catalog_manager.cc:671] Loaded metadata for table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9]
I20250626 14:04:35.550882 2334 catalog_manager.cc:671] Loaded metadata for table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]
I20250626 14:04:35.551035 2334 catalog_manager.cc:671] Loaded metadata for table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]
I20250626 14:04:35.551190 2334 catalog_manager.cc:671] Loaded metadata for table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]
I20250626 14:04:35.551333 2334 catalog_manager.cc:671] Loaded metadata for table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]
I20250626 14:04:35.551499 2334 catalog_manager.cc:671] Loaded metadata for table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]
I20250626 14:04:35.551728 2334 catalog_manager.cc:671] Loaded metadata for table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]
I20250626 14:04:35.552695 2334 tablet_loader.cc:96] loaded metadata for tablet 0db0b418e8b8412da3dc098ef6aab940 (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140])
I20250626 14:04:35.552978 2334 tablet_loader.cc:96] loaded metadata for tablet 5f5ce7fc87164ae1ace91557275285f8 (table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233])
I20250626 14:04:35.553117 2334 tablet_loader.cc:96] loaded metadata for tablet 680301fb5722406194f4dcadeb726964 (table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9])
I20250626 14:04:35.553496 2334 tablet_loader.cc:96] loaded metadata for tablet 8e965ba7276e469c94493334ba18fa1b (table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c])
I20250626 14:04:35.553625 2334 tablet_loader.cc:96] loaded metadata for tablet a4e9c6ce944b4ec6af44f7852cac93d3 (table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75])
I20250626 14:04:35.553836 2334 tablet_loader.cc:96] loaded metadata for tablet b25221169c9c48718276736fe657f378 (table default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843])
I20250626 14:04:35.553964 2334 tablet_loader.cc:96] loaded metadata for tablet b325108f7de949199cc10dbe259a0651 (table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb])
I20250626 14:04:35.554165 2334 tablet_loader.cc:96] loaded metadata for tablet d21f45117538444ba7b968411a58f099 (table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a])
I20250626 14:04:35.554371 2334 tablet_loader.cc:96] loaded metadata for tablet dd5f0eb6de154b3c9ffdd6d8f7042a49 (table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03])
I20250626 14:04:35.554500 2334 tablet_loader.cc:96] loaded metadata for tablet f51154f6fd584a95976d6af90f9133e9 (table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80])
I20250626 14:04:35.554628 2334 tablet_loader.cc:96] loaded metadata for tablet f7fa3619f0c541bf8e83db2d4fd24ced (table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447])
I20250626 14:04:35.554883 2334 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 14:04:35.555449 2334 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:35.555492 2334 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 14:04:35.556282 2334 catalog_manager.cc:1506] Loading token signing keys...
I20250626 14:04:35.556815 2334 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Loaded TSK: 0
I20250626 14:04:35.557296 2334 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 14:04:35.557781 2334 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250626 14:04:35.558264 2334 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 13
I20250626 14:04:35.587684 1478 master_service.cc:948] ReplaceTablet: received request to replace tablet f51154f6fd584a95976d6af90f9133e9 from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:35.587800 1477 master_service.cc:948] ReplaceTablet: received request to replace tablet f7fa3619f0c541bf8e83db2d4fd24ced from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:35.588403 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet f51154f6fd584a95976d6af90f9133e9 from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:35.588521 1481 master_service.cc:948] ReplaceTablet: received request to replace tablet 8e965ba7276e469c94493334ba18fa1b from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:35.588711 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_566f55c5d8c34043a37041fef8cb9f45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:35.589169 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_fdd6a46b24984290ac6f503f5684b003"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:35.589232 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet 8e965ba7276e469c94493334ba18fa1b from {username='slave'} at 127.0.0.1:51198
I20250626 14:04:35.589264 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_8c238e2636e9491f938a48294f961982"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:35.588655 1479 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_213d95d0e97546c298d94c45091826cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:35.589188 1480 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_2c8dee0163784a60a8468b53619247f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:35.592933 1477 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Sending DeleteTablet for 3 replicas of tablet f7fa3619f0c541bf8e83db2d4fd24ced
I20250626 14:04:35.593284 1477 catalog_manager.cc:6652] ReplaceTablet: tablet f7fa3619f0c541bf8e83db2d4fd24ced deleted and replaced by tablet 17a5b09fa0b54d45b835aa64dac8a1af
I20250626 14:04:35.593475 1477 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_22f5000aea364833a97948c4478f7e27" } new_table_name: "default.Table_8268b369fb20436fb4e562f6a97eee1d" modify_external_catalogs: true
I20250626 14:04:35.593923 1658 tablet_service.cc:1515] Processing DeleteTablet for tablet f7fa3619f0c541bf8e83db2d4fd24ced with delete_type TABLET_DATA_DELETED (replaced by tablet 17a5b09fa0b54d45b835aa64dac8a1af) from {username='slave'} at 127.0.0.1:43706
I20250626 14:04:35.594615 2365 tablet_replica.cc:331] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:35.594599 1792 tablet_service.cc:1515] Processing DeleteTablet for tablet f7fa3619f0c541bf8e83db2d4fd24ced with delete_type TABLET_DATA_DELETED (replaced by tablet 17a5b09fa0b54d45b835aa64dac8a1af) from {username='slave'} at 127.0.0.1:48162
I20250626 14:04:35.594905 2365 raft_consensus.cc:2241] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:35.594996 2368 tablet_replica.cc:331] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:35.595094 1928 tablet_service.cc:1515] Processing DeleteTablet for tablet f7fa3619f0c541bf8e83db2d4fd24ced with delete_type TABLET_DATA_DELETED (replaced by tablet 17a5b09fa0b54d45b835aa64dac8a1af) from {username='slave'} at 127.0.0.1:54640
I20250626 14:04:35.595132 2368 raft_consensus.cc:2241] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:35.595342 2368 raft_consensus.cc:2270] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:35.595439 2365 raft_consensus.cc:2270] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:35.595873 2365 ts_tablet_manager.cc:1905] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:35.595882 2368 ts_tablet_manager.cc:1905] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:35.596253 2369 tablet_replica.cc:331] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:35.596414 2369 raft_consensus.cc:2241] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:35.596596 2369 raft_consensus.cc:2270] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:35.596997 2369 ts_tablet_manager.cc:1905] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
F20250626 14:04:35.597708 1560 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 16, previous 16; dumping first 4 out of 8 received events: NotificationEvent(eventId=14, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_72f9307d2f5247a0ae4499c953ea1650, message={"server":"","servicePrincipal":"","db":"default","table":"table_72f9307d2f5247a0ae4499c953ea1650","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_72f9307d2f5247a0ae4499c953ea1650\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_72f9307d2f5247a0ae4499c953ea1650\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_72f9307d2f5247a0ae4499c953ea1650\",\"kudu.table_id\":\"61be84e63b2e478b9da5619fdbdffee9\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=15, eventTime=1750946674, eventType=CREATE_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=ALTER_TABLE, dbName=default, tableName=table_7d5a9b0c3ae440b89dac3ca5dd8e81ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_fb8ebc7aa8894f28b4d79252382f356a","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_fb8ebc7aa8894f28b4d79252382f356a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_fb8ebc7aa8894f28b4d79252382f356a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fb8ebc7aa8894f28b4d79252382f356a\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250626 14:04:35.598062 1481 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Sending DeleteTablet for 3 replicas of tablet 8e965ba7276e469c94493334ba18fa1b
I20250626 14:04:35.598405 2365 ts_tablet_manager.cc:1918] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:35.598491 2365 log.cc:1199] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/f7fa3619f0c541bf8e83db2d4fd24ced
I20250626 14:04:35.598479 1481 catalog_manager.cc:6652] ReplaceTablet: tablet 8e965ba7276e469c94493334ba18fa1b deleted and replaced by tablet 63672ab3fcd14a8783f3821d924c26bd
I20250626 14:04:35.598788 1658 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e965ba7276e469c94493334ba18fa1b with delete_type TABLET_DATA_DELETED (replaced by tablet 63672ab3fcd14a8783f3821d924c26bd) from {username='slave'} at 127.0.0.1:43706
I20250626 14:04:35.598788 1481 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51198:
name: "default.Table_33ee3521f64d4cefb8856259510e5f0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:35.599116 2365 ts_tablet_manager.cc:1939] T f7fa3619f0c541bf8e83db2d4fd24ced P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:35.599345 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_0b500f44e16e45e589c01ab7ad9a621b" } modify_external_catalogs: true
I20250626 14:04:35.599504 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_0b500f44e16e45e589c01ab7ad9a621b" } modify_external_catalogs: true
I20250626 14:04:35.599956 2365 tablet_replica.cc:331] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:35.599900 1473 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet f7fa3619f0c541bf8e83db2d4fd24ced (table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]) successfully deleted
I20250626 14:04:35.600262 2365 raft_consensus.cc:2241] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:35.600469 2365 raft_consensus.cc:2270] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:35.600648 2369 ts_tablet_manager.cc:1918] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:35.600715 2369 log.cc:1199] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/f7fa3619f0c541bf8e83db2d4fd24ced
I20250626 14:04:35.600853 1792 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e965ba7276e469c94493334ba18fa1b with delete_type TABLET_DATA_DELETED (replaced by tablet 63672ab3fcd14a8783f3821d924c26bd) from {username='slave'} at 127.0.0.1:48162
I20250626 14:04:35.600967 2369 ts_tablet_manager.cc:1939] T f7fa3619f0c541bf8e83db2d4fd24ced P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:35.601059 2365 ts_tablet_manager.cc:1905] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:35.601207 1928 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e965ba7276e469c94493334ba18fa1b with delete_type TABLET_DATA_DELETED (replaced by tablet 63672ab3fcd14a8783f3821d924c26bd) from {username='slave'} at 127.0.0.1:54640
@ 0x7f18d4e36dcd google::LogMessage::Fail() at ??:0
I20250626 14:04:35.601559 2369 tablet_replica.cc:331] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:35.601671 2369 raft_consensus.cc:2241] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:35.601768 2369 raft_consensus.cc:2270] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:35.602238 1472 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet f7fa3619f0c541bf8e83db2d4fd24ced (table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]) successfully deleted
I20250626 14:04:35.603008 2365 ts_tablet_manager.cc:1918] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:35.603070 2365 log.cc:1199] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/8e965ba7276e469c94493334ba18fa1b
I20250626 14:04:35.603310 2365 ts_tablet_manager.cc:1939] T 8e965ba7276e469c94493334ba18fa1b P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
@ 0x7f18d4e3ab93 google::LogMessage::SendToLog() at ??:0
I20250626 14:04:35.604315 2369 ts_tablet_manager.cc:1905] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:35.604585 1473 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet 8e965ba7276e469c94493334ba18fa1b (table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]) successfully deleted
I20250626 14:04:35.605048 1478 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Sending DeleteTablet for 3 replicas of tablet f51154f6fd584a95976d6af90f9133e9
I20250626 14:04:35.605348 1478 catalog_manager.cc:6652] ReplaceTablet: tablet f51154f6fd584a95976d6af90f9133e9 deleted and replaced by tablet 7c18e8a737f548c6ab77f739ca17135b
I20250626 14:04:35.605537 1478 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_42d07c2b16314b1999eea5f3878944fe" } modify_external_catalogs: true
I20250626 14:04:35.605672 1478 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_42d07c2b16314b1999eea5f3878944fe" } modify_external_catalogs: true
I20250626 14:04:35.606003 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51198:
table { table_name: "default.Table_afd26acd01c7422fbb06520af5a4f76c" } new_table_name: "default.Table_4d81e86bd6614c3db77efeaca0daa56f" modify_external_catalogs: true
@ 0x7f18d4e367cc google::LogMessage::Flush() at ??:0
I20250626 14:04:35.606174 2368 ts_tablet_manager.cc:1918] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:35.606256 2368 log.cc:1199] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/f7fa3619f0c541bf8e83db2d4fd24ced
I20250626 14:04:35.606452 2369 ts_tablet_manager.cc:1918] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:35.606519 2369 log.cc:1199] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/8e965ba7276e469c94493334ba18fa1b
I20250626 14:04:35.606556 2368 ts_tablet_manager.cc:1939] T f7fa3619f0c541bf8e83db2d4fd24ced P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:35.606580 1792 tablet_service.cc:1515] Processing DeleteTablet for tablet f51154f6fd584a95976d6af90f9133e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c18e8a737f548c6ab77f739ca17135b) from {username='slave'} at 127.0.0.1:48162
I20250626 14:04:35.606981 1658 tablet_service.cc:1515] Processing DeleteTablet for tablet f51154f6fd584a95976d6af90f9133e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c18e8a737f548c6ab77f739ca17135b) from {username='slave'} at 127.0.0.1:43706
I20250626 14:04:35.607204 2365 tablet_replica.cc:331] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:35.607229 2368 tablet_replica.cc:331] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:35.607331 2365 raft_consensus.cc:2241] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:35.607378 2368 raft_consensus.cc:2241] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:35.607440 2365 raft_consensus.cc:2270] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:35.607488 2368 raft_consensus.cc:2270] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:35.607571 1472 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet f7fa3619f0c541bf8e83db2d4fd24ced (table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]) successfully deleted
I20250626 14:04:35.607745 2369 ts_tablet_manager.cc:1939] T 8e965ba7276e469c94493334ba18fa1b P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:35.608022 2365 ts_tablet_manager.cc:1905] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f18d4e36a11 google::LogMessage::~LogMessage() at ??:0
I20250626 14:04:35.609035 2368 ts_tablet_manager.cc:1905] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:35.609112 1928 tablet_service.cc:1515] Processing DeleteTablet for tablet f51154f6fd584a95976d6af90f9133e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c18e8a737f548c6ab77f739ca17135b) from {username='slave'} at 127.0.0.1:54640
I20250626 14:04:35.609270 1472 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet 8e965ba7276e469c94493334ba18fa1b (table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]) successfully deleted
I20250626 14:04:35.609462 2369 tablet_replica.cc:331] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:35.609562 2369 raft_consensus.cc:2241] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:35.609730 2369 raft_consensus.cc:2270] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:35.610087 2369 ts_tablet_manager.cc:1905] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:35.610116 2365 ts_tablet_manager.cc:1918] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:35.610173 2365 log.cc:1199] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/f51154f6fd584a95976d6af90f9133e9
I20250626 14:04:35.610419 2365 ts_tablet_manager.cc:1939] T f51154f6fd584a95976d6af90f9133e9 P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:35.610972 1473 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet f51154f6fd584a95976d6af90f9133e9 (table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]) successfully deleted
I20250626 14:04:35.611513 2368 ts_tablet_manager.cc:1918] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:35.611603 2368 log.cc:1199] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/8e965ba7276e469c94493334ba18fa1b
I20250626 14:04:35.611909 2368 ts_tablet_manager.cc:1939] T 8e965ba7276e469c94493334ba18fa1b P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:35.612658 2368 tablet_replica.cc:331] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651: stopping tablet replica
@ 0x7f18d9dcfc91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250626 14:04:35.612794 2368 raft_consensus.cc:2241] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:35.612802 2369 ts_tablet_manager.cc:1918] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:35.612874 2369 log.cc:1199] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/f51154f6fd584a95976d6af90f9133e9
I20250626 14:04:35.612917 2368 raft_consensus.cc:2270] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:35.613101 2369 ts_tablet_manager.cc:1939] T f51154f6fd584a95976d6af90f9133e9 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:35.613044 1658 tablet_service.cc:1468] Processing CreateTablet for tablet 17a5b09fa0b54d45b835aa64dac8a1af (DEFAULT_TABLE table=default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:35.613301 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17a5b09fa0b54d45b835aa64dac8a1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:35.613312 2368 ts_tablet_manager.cc:1905] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:35.613874 1657 tablet_service.cc:1468] Processing CreateTablet for tablet b25221169c9c48718276736fe657f378 (DEFAULT_TABLE table=default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:35.614082 1657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25221169c9c48718276736fe657f378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:35.615777 1472 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet f51154f6fd584a95976d6af90f9133e9 (table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]) successfully deleted
I20250626 14:04:35.615895 1472 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet 8e965ba7276e469c94493334ba18fa1b (table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]) successfully deleted
I20250626 14:04:35.616158 1928 tablet_service.cc:1468] Processing CreateTablet for tablet 17a5b09fa0b54d45b835aa64dac8a1af (DEFAULT_TABLE table=default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:35.616444 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17a5b09fa0b54d45b835aa64dac8a1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:35.616418 1792 tablet_service.cc:1468] Processing CreateTablet for tablet 17a5b09fa0b54d45b835aa64dac8a1af (DEFAULT_TABLE table=default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:35.616690 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17a5b09fa0b54d45b835aa64dac8a1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:35.618516 1793 tablet_service.cc:1468] Processing CreateTablet for tablet b25221169c9c48718276736fe657f378 (DEFAULT_TABLE table=default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:35.618752 1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25221169c9c48718276736fe657f378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:35.619050 1927 tablet_service.cc:1468] Processing CreateTablet for tablet b25221169c9c48718276736fe657f378 (DEFAULT_TABLE table=default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:35.619316 1927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25221169c9c48718276736fe657f378. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f18d9dceba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250626 14:04:35.621420 2368 ts_tablet_manager.cc:1918] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:35.621481 2368 log.cc:1199] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/f51154f6fd584a95976d6af90f9133e9
I20250626 14:04:35.621740 2368 ts_tablet_manager.cc:1939] T f51154f6fd584a95976d6af90f9133e9 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:35.621884 2373 tablet_bootstrap.cc:492] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:35.622215 2372 tablet_bootstrap.cc:492] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:35.622435 2371 tablet_bootstrap.cc:492] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:35.622601 1472 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet f51154f6fd584a95976d6af90f9133e9 (table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]) successfully deleted
I20250626 14:04:35.622835 2373 tablet_bootstrap.cc:654] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:35.623361 2371 tablet_bootstrap.cc:654] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:35.623922 2373 tablet_bootstrap.cc:492] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:35.624006 2373 ts_tablet_manager.cc:1397] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:35.624615 2373 raft_consensus.cc:357] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.624742 2373 raft_consensus.cc:383] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:35.624783 2373 raft_consensus.cc:738] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:35.624924 2373 consensus_queue.cc:260] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.625069 1793 tablet_service.cc:1468] Processing CreateTablet for tablet 63672ab3fcd14a8783f3821d924c26bd (DEFAULT_TABLE table=default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:35.625272 2373 ts_tablet_manager.cc:1428] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 14:04:35.625315 1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63672ab3fcd14a8783f3821d924c26bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:35.625367 1657 tablet_service.cc:1468] Processing CreateTablet for tablet 63672ab3fcd14a8783f3821d924c26bd (DEFAULT_TABLE table=default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f18d9dce30b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250626 14:04:35.625519 2373 tablet_bootstrap.cc:492] T b25221169c9c48718276736fe657f378 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:35.625588 1657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63672ab3fcd14a8783f3821d924c26bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:35.626462 1792 tablet_service.cc:1468] Processing CreateTablet for tablet 7c18e8a737f548c6ab77f739ca17135b (DEFAULT_TABLE table=default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:35.626657 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c18e8a737f548c6ab77f739ca17135b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:35.626767 2373 tablet_bootstrap.cc:654] T b25221169c9c48718276736fe657f378 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:35.627427 1658 tablet_service.cc:1468] Processing CreateTablet for tablet 7c18e8a737f548c6ab77f739ca17135b (DEFAULT_TABLE table=default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:35.627653 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c18e8a737f548c6ab77f739ca17135b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:35.628362 1927 tablet_service.cc:1468] Processing CreateTablet for tablet 63672ab3fcd14a8783f3821d924c26bd (DEFAULT_TABLE table=default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f18d9dd3a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250626 14:04:35.628659 1927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63672ab3fcd14a8783f3821d924c26bd. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x55a49ba18752 std::function<>::operator()() at ??:0
I20250626 14:04:35.630831 2372 tablet_bootstrap.cc:654] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:35.631201 1928 tablet_service.cc:1468] Processing CreateTablet for tablet 7c18e8a737f548c6ab77f739ca17135b (DEFAULT_TABLE table=default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:35.631453 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c18e8a737f548c6ab77f739ca17135b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:35.631508 2371 tablet_bootstrap.cc:492] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:35.631596 2371 ts_tablet_manager.cc:1397] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250626 14:04:35.632210 2372 tablet_bootstrap.cc:492] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:35.632138 2371 raft_consensus.cc:357] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
@ 0x7f18d62880a1 kudu::Thread::SuperviseThread() at ??:0
I20250626 14:04:35.632247 2371 raft_consensus.cc:383] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:35.632308 2372 ts_tablet_manager.cc:1397] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250626 14:04:35.632308 2371 raft_consensus.cc:738] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:35.632440 2371 consensus_queue.cc:260] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.632807 2371 ts_tablet_manager.cc:1428] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:35.632830 2372 raft_consensus.cc:357] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.632928 2372 raft_consensus.cc:383] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:35.632961 2372 raft_consensus.cc:738] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:35.633018 2371 tablet_bootstrap.cc:492] T b25221169c9c48718276736fe657f378 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:35.633071 2372 consensus_queue.cc:260] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.633337 2372 ts_tablet_manager.cc:1428] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:35.633564 2372 tablet_bootstrap.cc:492] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:35.633952 2371 tablet_bootstrap.cc:654] T b25221169c9c48718276736fe657f378 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:35.634541 2372 tablet_bootstrap.cc:654] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
@ 0x7f18d6a776db start_thread at ??:0
@ 0x7f18d3fed71f clone at ??:0
I20250626 14:04:35.638190 2371 tablet_bootstrap.cc:492] T b25221169c9c48718276736fe657f378 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:35.638197 2373 tablet_bootstrap.cc:492] T b25221169c9c48718276736fe657f378 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:35.638298 2371 ts_tablet_manager.cc:1397] T b25221169c9c48718276736fe657f378 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250626 14:04:35.638316 2373 ts_tablet_manager.cc:1397] T b25221169c9c48718276736fe657f378 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250626 14:04:35.638804 2371 raft_consensus.cc:357] T b25221169c9c48718276736fe657f378 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.638909 2371 raft_consensus.cc:383] T b25221169c9c48718276736fe657f378 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:35.638870 2373 raft_consensus.cc:357] T b25221169c9c48718276736fe657f378 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.638950 2371 raft_consensus.cc:738] T b25221169c9c48718276736fe657f378 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:35.638986 2373 raft_consensus.cc:383] T b25221169c9c48718276736fe657f378 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:35.639022 2373 raft_consensus.cc:738] T b25221169c9c48718276736fe657f378 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:35.639194 2373 consensus_queue.cc:260] T b25221169c9c48718276736fe657f378 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.639513 2373 ts_tablet_manager.cc:1428] T b25221169c9c48718276736fe657f378 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:35.639477 2371 consensus_queue.cc:260] T b25221169c9c48718276736fe657f378 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.639725 2372 tablet_bootstrap.cc:492] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:35.639789 2371 ts_tablet_manager.cc:1428] T b25221169c9c48718276736fe657f378 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:35.639699 2373 tablet_bootstrap.cc:492] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:35.639806 2372 ts_tablet_manager.cc:1397] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250626 14:04:35.639982 2371 tablet_bootstrap.cc:492] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:35.640287 2372 raft_consensus.cc:357] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.640398 2372 raft_consensus.cc:383] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:35.640446 2372 raft_consensus.cc:738] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:35.640630 2372 consensus_queue.cc:260] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.640903 2372 ts_tablet_manager.cc:1428] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:35.641054 2371 tablet_bootstrap.cc:654] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:35.641069 2373 tablet_bootstrap.cc:654] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:35.641104 2372 tablet_bootstrap.cc:492] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:35.642190 2372 tablet_bootstrap.cc:654] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:35.644094 2373 tablet_bootstrap.cc:492] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:35.644176 2373 ts_tablet_manager.cc:1397] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250626 14:04:35.644698 2373 raft_consensus.cc:357] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.644810 2373 raft_consensus.cc:383] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:35.644848 2373 raft_consensus.cc:738] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:35.644982 2371 tablet_bootstrap.cc:492] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:35.644977 2373 consensus_queue.cc:260] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.645054 2371 ts_tablet_manager.cc:1397] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20250626 14:04:35.645259 2373 ts_tablet_manager.cc:1428] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:35.645463 2371 raft_consensus.cc:357] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.645553 2371 raft_consensus.cc:383] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:35.645582 2371 raft_consensus.cc:738] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:35.645692 2371 consensus_queue.cc:260] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.645910 2371 ts_tablet_manager.cc:1428] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:35.646059 2371 tablet_bootstrap.cc:492] T 63672ab3fcd14a8783f3821d924c26bd P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:35.647099 2371 tablet_bootstrap.cc:654] T 63672ab3fcd14a8783f3821d924c26bd P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:35.649363 2371 tablet_bootstrap.cc:492] T 63672ab3fcd14a8783f3821d924c26bd P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:35.649441 2371 ts_tablet_manager.cc:1397] T 63672ab3fcd14a8783f3821d924c26bd P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250626 14:04:35.649931 2371 raft_consensus.cc:357] T 63672ab3fcd14a8783f3821d924c26bd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.650040 2371 raft_consensus.cc:383] T 63672ab3fcd14a8783f3821d924c26bd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:35.650084 2371 raft_consensus.cc:738] T 63672ab3fcd14a8783f3821d924c26bd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:35.650199 2371 consensus_queue.cc:260] T 63672ab3fcd14a8783f3821d924c26bd P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.650530 2371 ts_tablet_manager.cc:1428] T 63672ab3fcd14a8783f3821d924c26bd P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:35.650607 2372 tablet_bootstrap.cc:492] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:35.650851 2372 ts_tablet_manager.cc:1397] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20250626 14:04:35.651351 2372 raft_consensus.cc:357] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.651463 2372 raft_consensus.cc:383] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:35.651508 2372 raft_consensus.cc:738] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:35.651682 2372 consensus_queue.cc:260] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.652729 2372 ts_tablet_manager.cc:1428] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 14:04:35.657227 2372 tablet_bootstrap.cc:492] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:35.658342 2373 tablet_bootstrap.cc:492] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:35.658699 2372 tablet_bootstrap.cc:654] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:35.659873 2372 tablet_bootstrap.cc:492] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:35.659950 2372 ts_tablet_manager.cc:1397] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250626 14:04:35.660446 2372 raft_consensus.cc:357] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.660591 2372 raft_consensus.cc:383] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:35.660637 2372 raft_consensus.cc:738] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:35.660789 2372 consensus_queue.cc:260] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.661682 2372 ts_tablet_manager.cc:1428] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 14:04:35.662087 2373 tablet_bootstrap.cc:654] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:35.663756 2373 tablet_bootstrap.cc:492] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:35.663841 2373 ts_tablet_manager.cc:1397] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250626 14:04:35.664302 2373 raft_consensus.cc:357] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.664414 2373 raft_consensus.cc:383] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:35.664445 2373 raft_consensus.cc:738] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:35.664602 2373 consensus_queue.cc:260] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.664884 2373 ts_tablet_manager.cc:1428] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:35.665693 2343 raft_consensus.cc:491] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:35.665782 2343 raft_consensus.cc:513] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.666219 2343 leader_election.cc:290] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:35.666558 2343 raft_consensus.cc:491] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:35.666621 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c18e8a737f548c6ab77f739ca17135b" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:35.666651 2343 raft_consensus.cc:513] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.666757 1677 raft_consensus.cc:2466] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:35.666990 2343 leader_election.cc:290] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:35.667001 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c18e8a737f548c6ab77f739ca17135b" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:35.667110 1947 raft_consensus.cc:2466] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:35.667485 1748 leader_election.cc:304] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:35.667692 2343 raft_consensus.cc:2802] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:35.667735 2343 raft_consensus.cc:491] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:35.667774 2343 raft_consensus.cc:3058] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:35.667930 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b25221169c9c48718276736fe657f378" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:35.668031 1947 raft_consensus.cc:2466] T b25221169c9c48718276736fe657f378 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:35.668427 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b25221169c9c48718276736fe657f378" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:35.668584 1677 raft_consensus.cc:2466] T b25221169c9c48718276736fe657f378 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:35.668874 2343 raft_consensus.cc:513] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.669210 2343 leader_election.cc:290] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Requested vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:35.668906 1749 leader_election.cc:304] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:35.669899 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c18e8a737f548c6ab77f739ca17135b" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:35.669910 2069 raft_consensus.cc:2802] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:35.669968 2069 raft_consensus.cc:491] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:35.669992 1947 raft_consensus.cc:3058] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:35.670003 2069 raft_consensus.cc:3058] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:35.670871 2069 raft_consensus.cc:513] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.671042 1947 raft_consensus.cc:2466] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:35.671162 2069 leader_election.cc:290] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Requested vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:35.671429 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c18e8a737f548c6ab77f739ca17135b" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:35.671546 1677 raft_consensus.cc:3058] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:35.671558 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b25221169c9c48718276736fe657f378" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:35.671730 1678 raft_consensus.cc:3058] T b25221169c9c48718276736fe657f378 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:35.672586 1677 raft_consensus.cc:2466] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:35.672691 1678 raft_consensus.cc:2466] T b25221169c9c48718276736fe657f378 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:35.673031 1748 leader_election.cc:304] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:35.673223 1948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b25221169c9c48718276736fe657f378" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:35.673267 1749 leader_election.cc:304] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:35.673322 1948 raft_consensus.cc:3058] T b25221169c9c48718276736fe657f378 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:35.673391 2069 raft_consensus.cc:2802] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:35.673449 2069 raft_consensus.cc:695] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Becoming Leader. State: Replica: e517893e13f546c0b40d976633b49651, State: Running, Role: LEADER
I20250626 14:04:35.673563 2069 consensus_queue.cc:237] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.674139 2069 raft_consensus.cc:2802] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:35.674198 2069 raft_consensus.cc:695] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Becoming Leader. State: Replica: e517893e13f546c0b40d976633b49651, State: Running, Role: LEADER
I20250626 14:04:35.674288 1948 raft_consensus.cc:2466] T b25221169c9c48718276736fe657f378 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:35.674288 2069 consensus_queue.cc:237] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.806034 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.254:36707
--webserver_interface=127.1.11.254
--webserver_port=45241
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250626 14:04:35.812239 2001 connection.cc:537] client connection to 127.1.11.252:37925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:35.812760 2001 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.11.252:37925)
W20250626 14:04:35.813169 2001 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.11.252:37925)
W20250626 14:04:35.813268 2001 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.11.252:37925)
W20250626 14:04:35.813437 2001 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.11.252:37925)
I20250626 14:04:35.825620 2066 raft_consensus.cc:491] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:35.825760 2066 raft_consensus.cc:513] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.826231 2066 leader_election.cc:290] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:35.826638 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17a5b09fa0b54d45b835aa64dac8a1af" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:35.826658 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17a5b09fa0b54d45b835aa64dac8a1af" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:35.826778 1813 raft_consensus.cc:2466] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 0.
I20250626 14:04:35.826797 1947 raft_consensus.cc:2466] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 0.
I20250626 14:04:35.827105 1613 leader_election.cc:304] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:35.827281 2066 raft_consensus.cc:2802] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:35.827342 2066 raft_consensus.cc:491] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:35.827378 2066 raft_consensus.cc:3058] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:35.828410 2066 raft_consensus.cc:513] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.828801 2066 leader_election.cc:290] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 election: Requested vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:35.829170 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17a5b09fa0b54d45b835aa64dac8a1af" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:35.829192 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17a5b09fa0b54d45b835aa64dac8a1af" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:35.829283 1947 raft_consensus.cc:3058] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:35.829290 1813 raft_consensus.cc:3058] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:35.830199 1813 raft_consensus.cc:2466] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
I20250626 14:04:35.830235 1947 raft_consensus.cc:2466] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
I20250626 14:04:35.830535 1613 leader_election.cc:304] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:35.830825 2066 raft_consensus.cc:2802] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:35.830906 2066 raft_consensus.cc:695] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [term 1 LEADER]: Becoming Leader. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Running, Role: LEADER
I20250626 14:04:35.831027 2066 consensus_queue.cc:237] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
W20250626 14:04:35.925554 2399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:35.925793 2399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:35.925837 2399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:04:35.929040 2399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:04:35.929096 2399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:35.929117 2399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:04:35.929136 2399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:04:35.933413 2399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.254:36707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.11.254
--webserver_port=45241
--never_fsync=true
--heap_profile_path=/tmp/kudu.2399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:35.934535 2399 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:35.935748 2399 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 14:04:35.939373 2069 raft_consensus.cc:491] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:35.939512 2069 raft_consensus.cc:513] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.940023 2069 leader_election.cc:290] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:35.940507 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63672ab3fcd14a8783f3821d924c26bd" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:35.940583 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63672ab3fcd14a8783f3821d924c26bd" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:35.940713 1947 raft_consensus.cc:2466] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:35.940726 1678 raft_consensus.cc:2466] T 63672ab3fcd14a8783f3821d924c26bd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:35.941006 1749 leader_election.cc:304] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:35.941180 2069 raft_consensus.cc:2802] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:35.941246 2069 raft_consensus.cc:491] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:35.941293 2069 raft_consensus.cc:3058] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:35.942479 2069 raft_consensus.cc:513] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.942965 2069 leader_election.cc:290] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Requested vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:04:35.942999 2407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:35.942981 2409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:35.943298 2399 server_base.cc:1048] running on GCE node
I20250626 14:04:35.943373 1947 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63672ab3fcd14a8783f3821d924c26bd" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:35.943496 1947 raft_consensus.cc:3058] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:35.944198 1947 raft_consensus.cc:2466] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:35.944442 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63672ab3fcd14a8783f3821d924c26bd" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:35.944604 1678 raft_consensus.cc:3058] T 63672ab3fcd14a8783f3821d924c26bd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:35.944600 1748 leader_election.cc:304] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:35.944861 2069 raft_consensus.cc:2802] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:35.944957 2069 raft_consensus.cc:695] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Becoming Leader. State: Replica: e517893e13f546c0b40d976633b49651, State: Running, Role: LEADER
I20250626 14:04:35.945094 2069 consensus_queue.cc:237] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:35.945462 1678 raft_consensus.cc:2466] T 63672ab3fcd14a8783f3821d924c26bd P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
W20250626 14:04:35.945737 2406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:35.946419 2399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:35.947517 2399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:35.948679 2399 hybrid_clock.cc:648] HybridClock initialized: now 1750946675948648 us; error 52 us; skew 500 ppm
I20250626 14:04:35.956504 2399 webserver.cc:469] Webserver started at http://127.1.11.254:45241/ using document root <none> and password file <none>
I20250626 14:04:35.957141 2399 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:35.957213 2399 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:35.960835 2399 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20250626 14:04:35.963047 2416 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:35.964107 2399 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:35.964211 2399 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
uuid: "24ad911d71b6433f8c7939a1d1f7b782"
format_stamp: "Formatted at 2025-06-26 14:04:30 on dist-test-slave-rptj"
I20250626 14:04:35.964763 2399 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:04:35.980755 2399 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:35.981370 2399 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:35.981520 2399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:35.989166 2399 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.254:36707
I20250626 14:04:35.989176 2468 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.254:36707 every 8 connection(s)
I20250626 14:04:35.990278 2399 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/info.pb
I20250626 14:04:35.994115 2469 sys_catalog.cc:263] Verifying existing consensus state
I20250626 14:04:35.995703 2469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Bootstrap starting.
I20250626 14:04:35.996654 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 2399
I20250626 14:04:36.007560 2469 log.cc:826] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Log is configured to *not* fsync() on all Append() calls
I20250626 14:04:36.041721 2469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Bootstrap replayed 1/1 log segments. Stats: ops{read=57 overwritten=0 applied=56 ignored=0} inserts{seen=39 ignored=0} mutations{seen=44 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250626 14:04:36.042210 2469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Bootstrap complete.
I20250626 14:04:36.045002 2469 raft_consensus.cc:357] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:36.046458 2469 raft_consensus.cc:738] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ad911d71b6433f8c7939a1d1f7b782, State: Initialized, Role: FOLLOWER
I20250626 14:04:36.046926 2469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 56, Last appended: 2.57, Last appended by leader: 57, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:36.047310 2475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:36.047415 2475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:36.048017 2469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:04:36.054720 2489 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:36.054795 2489 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: loading cluster ID for follower catalog manager: success
I20250626 14:04:36.057896 2489 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: acquiring CA information for follower catalog manager: success
I20250626 14:04:36.059912 2489 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 14:04:36.066216 2399 master_runner.cc:186] Error getting master registration for 127.1.11.252:37925: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111),
I20250626 14:04:36.068300 2399 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:04:36.117434 2069 consensus_queue.cc:1035] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:36.123132 2325 consensus_queue.cc:1035] T b25221169c9c48718276736fe657f378 P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:36.127866 2501 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 81db61eed5584e7e8b0cf40e5441f88e)
I20250626 14:04:36.127977 2501 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:36.129106 2501 leader_election.cc:290] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
I20250626 14:04:36.129419 1995 heartbeater.cc:344] Connected to a master server at 127.1.11.254:36707
I20250626 14:04:36.131325 1725 heartbeater.cc:344] Connected to a master server at 127.1.11.254:36707
I20250626 14:04:36.131356 1860 heartbeater.cc:344] Connected to a master server at 127.1.11.254:36707
I20250626 14:04:36.132846 2431 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" instance_seqno: 1750946672002466) as {username='slave'} at 127.1.11.195:44997; Asking this server to re-register.
I20250626 14:04:36.133266 2433 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e517893e13f546c0b40d976633b49651" instance_seqno: 1750946671821196) as {username='slave'} at 127.1.11.194:55581; Asking this server to re-register.
I20250626 14:04:36.133417 2432 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" instance_seqno: 1750946671637378) as {username='slave'} at 127.1.11.193:51365; Asking this server to re-register.
I20250626 14:04:36.133951 1995 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:36.133949 1860 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:36.134708 1725 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:36.135707 2433 ts_manager.cc:194] Registered new tserver with Master: 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:36.136317 2443 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07532b7648e4414789ed2b099bad2b19" candidate_term: 4 candidate_status { last_received { term: 3 index: 63 } } ignore_live_leader: false dest_uuid: "24ad911d71b6433f8c7939a1d1f7b782" is_pre_election: true
I20250626 14:04:36.136446 2432 ts_manager.cc:194] Registered new tserver with Master: e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:36.136703 2443 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07532b7648e4414789ed2b099bad2b19 in term 2.
I20250626 14:04:36.137043 2431 ts_manager.cc:194] Registered new tserver with Master: 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:36.137069 2124 leader_election.cc:304] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782; no voters:
I20250626 14:04:36.137244 2501 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 14:04:36.137291 2501 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 81db61eed5584e7e8b0cf40e5441f88e)
I20250626 14:04:36.137336 2501 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 3 FOLLOWER]: Advancing to term 4
W20250626 14:04:36.138279 2124 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:04:36.138473 2501 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:36.138840 2501 leader_election.cc:290] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 4 election: Requested vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
I20250626 14:04:36.139115 2443 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07532b7648e4414789ed2b099bad2b19" candidate_term: 4 candidate_status { last_received { term: 3 index: 63 } } ignore_live_leader: false dest_uuid: "24ad911d71b6433f8c7939a1d1f7b782"
I20250626 14:04:36.139242 2443 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 2 FOLLOWER]: Advancing to term 4
W20250626 14:04:36.139771 2124 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:04:36.141399 2443 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07532b7648e4414789ed2b099bad2b19 in term 4.
I20250626 14:04:36.141741 2124 leader_election.cc:304] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782; no voters: 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:04:36.141925 2501 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 14:04:36.142138 2501 raft_consensus.cc:695] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 4 LEADER]: Becoming Leader. State: Replica: 07532b7648e4414789ed2b099bad2b19, State: Running, Role: LEADER
I20250626 14:04:36.142444 2501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 3.63, Last appended by leader: 63, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:36.143374 2507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07532b7648e4414789ed2b099bad2b19. Latest consensus state: current_term: 4 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:36.143481 2507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:36.143815 2508 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 14:04:36.145968 2508 catalog_manager.cc:671] Loaded metadata for table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]
I20250626 14:04:36.146293 2508 catalog_manager.cc:671] Loaded metadata for table default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843]
I20250626 14:04:36.146548 2508 catalog_manager.cc:671] Loaded metadata for table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]
I20250626 14:04:36.146899 2508 catalog_manager.cc:671] Loaded metadata for table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]
I20250626 14:04:36.147136 2508 catalog_manager.cc:671] Loaded metadata for table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9]
I20250626 14:04:36.147356 2508 catalog_manager.cc:671] Loaded metadata for table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]
I20250626 14:04:36.147614 2508 catalog_manager.cc:671] Loaded metadata for table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]
I20250626 14:04:36.147872 2508 catalog_manager.cc:671] Loaded metadata for table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]
I20250626 14:04:36.148119 2508 catalog_manager.cc:671] Loaded metadata for table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]
I20250626 14:04:36.148334 2508 catalog_manager.cc:671] Loaded metadata for table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]
I20250626 14:04:36.148715 2508 catalog_manager.cc:671] Loaded metadata for table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]
I20250626 14:04:36.150326 2508 tablet_loader.cc:96] loaded metadata for tablet 0db0b418e8b8412da3dc098ef6aab940 (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140])
I20250626 14:04:36.150566 2508 tablet_loader.cc:96] loaded metadata for tablet 17a5b09fa0b54d45b835aa64dac8a1af (table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447])
I20250626 14:04:36.150876 2508 tablet_loader.cc:96] loaded metadata for tablet 5f5ce7fc87164ae1ace91557275285f8 (table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233])
I20250626 14:04:36.151063 2508 tablet_loader.cc:96] loaded metadata for tablet 63672ab3fcd14a8783f3821d924c26bd (table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c])
I20250626 14:04:36.151242 2508 tablet_loader.cc:96] loaded metadata for tablet 680301fb5722406194f4dcadeb726964 (table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9])
I20250626 14:04:36.151952 2508 tablet_loader.cc:96] loaded metadata for tablet 7c18e8a737f548c6ab77f739ca17135b (table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80])
I20250626 14:04:36.152297 2508 tablet_loader.cc:96] loaded metadata for tablet a4e9c6ce944b4ec6af44f7852cac93d3 (table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75])
I20250626 14:04:36.152657 2508 tablet_loader.cc:96] loaded metadata for tablet b25221169c9c48718276736fe657f378 (table default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843])
I20250626 14:04:36.153034 2508 tablet_loader.cc:96] loaded metadata for tablet b325108f7de949199cc10dbe259a0651 (table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb])
I20250626 14:04:36.153348 2508 tablet_loader.cc:96] loaded metadata for tablet d21f45117538444ba7b968411a58f099 (table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a])
I20250626 14:04:36.153694 2508 tablet_loader.cc:96] loaded metadata for tablet dd5f0eb6de154b3c9ffdd6d8f7042a49 (table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03])
I20250626 14:04:36.154165 2508 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 14:04:36.154955 2508 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:36.155014 2508 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 14:04:36.155946 2508 catalog_manager.cc:1506] Loading token signing keys...
I20250626 14:04:36.156672 2508 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Loaded TSK: 0
I20250626 14:04:36.157565 2508 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 14:04:36.158272 2508 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250626 14:04:36.158949 2508 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 13
I20250626 14:04:36.181990 2137 master_service.cc:948] ReplaceTablet: received request to replace tablet a4e9c6ce944b4ec6af44f7852cac93d3 from {username='slave'} at 127.0.0.1:49330
I20250626 14:04:36.183339 2140 master_service.cc:948] ReplaceTablet: received request to replace tablet d21f45117538444ba7b968411a58f099 from {username='slave'} at 127.0.0.1:49330
I20250626 14:04:36.183758 2135 master_service.cc:948] ReplaceTablet: received request to replace tablet 680301fb5722406194f4dcadeb726964 from {username='slave'} at 127.0.0.1:49330
I20250626 14:04:36.183862 2138 master_service.cc:948] ReplaceTablet: received request to replace tablet 63672ab3fcd14a8783f3821d924c26bd from {username='slave'} at 127.0.0.1:49330
I20250626 14:04:36.184242 2136 master_service.cc:948] ReplaceTablet: received request to replace tablet b325108f7de949199cc10dbe259a0651 from {username='slave'} at 127.0.0.1:49330
I20250626 14:04:36.183140 2131 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49330:
table { table_name: "default.Table_f9b5b5e2dcf64f9e889c618242274d25" } modify_external_catalogs: true
I20250626 14:04:36.183369 2132 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49330:
name: "default.Table_5bf40f09daaa4240a8bcab30e1aa341f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:36.184916 2131 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49330:
table { table_name: "default.Table_f9b5b5e2dcf64f9e889c618242274d25" } modify_external_catalogs: true
I20250626 14:04:36.183688 2134 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49330:
name: "default.Table_f7012cd4bf5d40d6af93cf9b01405dcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:36.183554 2133 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49330:
table { table_name: "default.Table_3a3f21b3c0e54e438561e02ab79fa36b" } new_table_name: "default.Table_3633fc38c5374180a081fab63a1eed84" modify_external_catalogs: true
I20250626 14:04:36.182974 2139 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49330:
table { table_name: "default.Table_e6e7d4c291af4d29baf43e204c5292f0" } new_table_name: "default.Table_f01c05221e5c4e75bf700b9474072881" modify_external_catalogs: true
W20250626 14:04:36.189043 2124 consensus_peers.cc:487] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 -> Peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Couldn't send request to peer 81db61eed5584e7e8b0cf40e5441f88e. Status: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 14:04:36.190934 2443 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Refusing update from remote peer 07532b7648e4414789ed2b099bad2b19: Log matching property violated. Preceding OpId in replica: term: 2 index: 57. Preceding OpId from leader: term: 4 index: 66. (index mismatch)
I20250626 14:04:36.191531 2501 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 56, Time since last communication: 0.000s
I20250626 14:04:36.193050 2325 consensus_queue.cc:1035] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
F20250626 14:04:36.195293 2199 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 16, previous 16; dumping first 4 out of 8 received events: NotificationEvent(eventId=14, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_72f9307d2f5247a0ae4499c953ea1650, message={"server":"","servicePrincipal":"","db":"default","table":"table_72f9307d2f5247a0ae4499c953ea1650","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_72f9307d2f5247a0ae4499c953ea1650\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_72f9307d2f5247a0ae4499c953ea1650\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_72f9307d2f5247a0ae4499c953ea1650\",\"kudu.table_id\":\"61be84e63b2e478b9da5619fdbdffee9\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=15, eventTime=1750946674, eventType=CREATE_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=ALTER_TABLE, dbName=default, tableName=table_7d5a9b0c3ae440b89dac3ca5dd8e81ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_fb8ebc7aa8894f28b4d79252382f356a","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_fb8ebc7aa8894f28b4d79252382f356a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_fb8ebc7aa8894f28b4d79252382f356a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fb8ebc7aa8894f28b4d79252382f356a\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f31a036bdcd google::LogMessage::Fail() at ??:0
I20250626 14:04:36.200001 2475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07532b7648e4414789ed2b099bad2b19. Latest consensus state: current_term: 4 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:36.200115 2475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:36.200212 2475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:36.200277 2475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f31a036fb93 google::LogMessage::SendToLog() at ??:0
I20250626 14:04:36.204339 2069 consensus_queue.cc:1035] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:36.205907 2474 mvcc.cc:204] Tried to move back new op lower bound from 7171877583336230912 to 7171877580399972352. Current Snapshot: MvccSnapshot[applied={T|T < 7171877583336230912}]
I20250626 14:04:36.207114 2511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:36.207237 2511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:36.209190 2516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
@ 0x7f31a036b7cc google::LogMessage::Flush() at ??:0
I20250626 14:04:36.209293 2516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:36.211707 2135 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Sending DeleteTablet for 3 replicas of tablet 680301fb5722406194f4dcadeb726964
@ 0x7f31a036ba11 google::LogMessage::~LogMessage() at ??:0
I20250626 14:04:36.212455 2066 consensus_queue.cc:1035] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:36.212957 2135 catalog_manager.cc:6652] ReplaceTablet: tablet 680301fb5722406194f4dcadeb726964 deleted and replaced by tablet 730ed7238006463eb08803a70330a43f
I20250626 14:04:36.214162 2135 catalog_manager.cc:5582] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b reported cstate change: term changed from 0 to 1, leader changed from <none> to e517893e13f546c0b40d976633b49651 (127.1.11.194). New cstate: current_term: 1 leader_uuid: "e517893e13f546c0b40d976633b49651" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } }
@ 0x7f31a5304c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250626 14:04:36.219475 2137 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Sending DeleteTablet for 3 replicas of tablet a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:04:36.219797 2137 catalog_manager.cc:6652] ReplaceTablet: tablet a4e9c6ce944b4ec6af44f7852cac93d3 deleted and replaced by tablet 33d002d179974a95aaad3d84814d27f0
@ 0x7f31a5303ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250626 14:04:36.222501 2138 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Sending DeleteTablet for 3 replicas of tablet 63672ab3fcd14a8783f3821d924c26bd
I20250626 14:04:36.222786 2138 catalog_manager.cc:6652] ReplaceTablet: tablet 63672ab3fcd14a8783f3821d924c26bd deleted and replaced by tablet f1c96426790242fdb8224137d55939f2
@ 0x7f31a530330b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250626 14:04:36.225309 2135 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49330:
table { table_name: "default.Table_fb8ebc7aa8894f28b4d79252382f356a" } new_table_name: "default.Table_7d5a9b0c3ae440b89dac3ca5dd8e81ef" modify_external_catalogs: true
I20250626 14:04:36.225548 1726 heartbeater.cc:499] Master 127.1.11.253:44447 was elected leader, sending a full tablet report...
I20250626 14:04:36.225682 2140 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Sending DeleteTablet for 3 replicas of tablet d21f45117538444ba7b968411a58f099
I20250626 14:04:36.225970 2140 catalog_manager.cc:6652] ReplaceTablet: tablet d21f45117538444ba7b968411a58f099 deleted and replaced by tablet 025415f7a05746e19d5ed199246fc949
I20250626 14:04:36.226166 2140 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49330:
name: "default.Table_f98cc740eae3444b88bb688eb8c8a316"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:36.227476 2136 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Sending DeleteTablet for 3 replicas of tablet b325108f7de949199cc10dbe259a0651
I20250626 14:04:36.227748 2136 catalog_manager.cc:6652] ReplaceTablet: tablet b325108f7de949199cc10dbe259a0651 deleted and replaced by tablet 1e307e729d7d42a68807263959c223a6
@ 0x7f31a5308a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x563bbcdd7752 std::function<>::operator()() at ??:0
I20250626 14:04:36.228664 2288 consensus_queue.cc:1035] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0x7f31a17bd0a1 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f31a1fac6db start_thread at ??:0
I20250626 14:04:36.234068 1928 tablet_service.cc:1515] Processing DeleteTablet for tablet 680301fb5722406194f4dcadeb726964 with delete_type TABLET_DATA_DELETED (replaced by tablet 730ed7238006463eb08803a70330a43f) from {username='slave'} at 127.0.0.1:38940
I20250626 14:04:36.234346 1926 tablet_service.cc:1515] Processing DeleteTablet for tablet 63672ab3fcd14a8783f3821d924c26bd with delete_type TABLET_DATA_DELETED (replaced by tablet f1c96426790242fdb8224137d55939f2) from {username='slave'} at 127.0.0.1:38940
I20250626 14:04:36.234493 1925 tablet_service.cc:1515] Processing DeleteTablet for tablet d21f45117538444ba7b968411a58f099 with delete_type TABLET_DATA_DELETED (replaced by tablet 025415f7a05746e19d5ed199246fc949) from {username='slave'} at 127.0.0.1:38940
I20250626 14:04:36.234609 1923 tablet_service.cc:1515] Processing DeleteTablet for tablet b325108f7de949199cc10dbe259a0651 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e307e729d7d42a68807263959c223a6) from {username='slave'} at 127.0.0.1:38940
I20250626 14:04:36.234079 1927 tablet_service.cc:1515] Processing DeleteTablet for tablet a4e9c6ce944b4ec6af44f7852cac93d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 33d002d179974a95aaad3d84814d27f0) from {username='slave'} at 127.0.0.1:38940
I20250626 14:04:36.234934 2532 tablet_replica.cc:331] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:36.235096 2532 raft_consensus.cc:2241] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:36.235236 2532 raft_consensus.cc:2270] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:36.235755 2532 ts_tablet_manager.cc:1905] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f319f52271f clone at ??:0
I20250626 14:04:36.238819 2532 ts_tablet_manager.cc:1918] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:36.238894 2532 log.cc:1199] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/680301fb5722406194f4dcadeb726964
I20250626 14:04:36.239162 2532 ts_tablet_manager.cc:1939] T 680301fb5722406194f4dcadeb726964 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:36.239894 2532 tablet_replica.cc:331] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:36.240012 2532 raft_consensus.cc:2241] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:36.240105 2532 raft_consensus.cc:2270] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:36.240446 2532 ts_tablet_manager.cc:1905] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:36.242532 2532 ts_tablet_manager.cc:1918] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 14:04:36.242599 2532 log.cc:1199] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/63672ab3fcd14a8783f3821d924c26bd
I20250626 14:04:36.242823 2532 ts_tablet_manager.cc:1939] T 63672ab3fcd14a8783f3821d924c26bd P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:36.243570 2532 tablet_replica.cc:331] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:36.243697 2532 raft_consensus.cc:2241] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:36.243811 2532 raft_consensus.cc:2270] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:36.244166 2532 ts_tablet_manager.cc:1905] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:36.246346 2532 ts_tablet_manager.cc:1918] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:36.246541 2532 log.cc:1199] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/d21f45117538444ba7b968411a58f099
I20250626 14:04:36.246856 2532 ts_tablet_manager.cc:1939] T d21f45117538444ba7b968411a58f099 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:36.247526 2532 tablet_replica.cc:331] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:36.247642 2532 raft_consensus.cc:2241] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:36.247850 2532 raft_consensus.cc:2270] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:36.248512 2532 ts_tablet_manager.cc:1905] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:36.250711 2532 ts_tablet_manager.cc:1918] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:36.250779 2532 log.cc:1199] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/b325108f7de949199cc10dbe259a0651
I20250626 14:04:36.251061 2532 ts_tablet_manager.cc:1939] T b325108f7de949199cc10dbe259a0651 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:36.251693 2532 tablet_replica.cc:331] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:36.251811 2532 raft_consensus.cc:2241] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:36.251919 2532 raft_consensus.cc:2270] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:36.252315 2532 ts_tablet_manager.cc:1905] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:36.254359 2532 ts_tablet_manager.cc:1918] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:36.254496 2532 log.cc:1199] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:04:36.254773 2532 ts_tablet_manager.cc:1939] T a4e9c6ce944b4ec6af44f7852cac93d3 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
W20250626 14:04:36.347553 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 1: this message will repeat every 5th retry.
W20250626 14:04:36.398563 1884 connection.cc:537] server connection from 127.0.0.1:38940 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:36.398865 2526 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 14:04:36.215362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.1.11.194:44397)
0626 14:04:36.238015 (+ 22653us) server_negotiation.cc:206] Beginning negotiation
0626 14:04:36.238021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0626 14:04:36.238040 (+ 19us) server_negotiation.cc:407] Connection header received
0626 14:04:36.238103 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0626 14:04:36.238107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0626 14:04:36.238186 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0626 14:04:36.238273 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0626 14:04:36.398705 (+160432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50928: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":22538,"thread_start_us":95,"threads_started":1}
W20250626 14:04:36.399868 2529 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 14:04:36.216831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.1.11.193:35671)
0626 14:04:36.230730 (+ 13899us) server_negotiation.cc:206] Beginning negotiation
0626 14:04:36.230736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0626 14:04:36.230750 (+ 14us) server_negotiation.cc:407] Connection header received
0626 14:04:36.230794 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0626 14:04:36.230797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0626 14:04:36.231916 (+ 1119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0626 14:04:36.232028 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0626 14:04:36.232616 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0626 14:04:36.236968 (+ 4352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0626 14:04:36.399726 (+162758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34706: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":13819,"thread_start_us":70,"threads_started":1}
W20250626 14:04:36.466620 1748 consensus_peers.cc:487] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3. This is attempt 1: this message will repeat every 5th retry.
W20250626 14:04:36.514416 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 1: this message will repeat every 5th retry.
W20250626 14:04:36.531140 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 1: this message will repeat every 5th retry.
I20250626 14:04:36.567730 2343 consensus_queue.cc:1035] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:36.736213 2547 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 07532b7648e4414789ed2b099bad2b19)
I20250626 14:04:36.736335 2547 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:36.737522 2547 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:04:36.737972 2419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:04:36.738632 2419 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:04:36.738736 2417 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:04:36.738842 2417 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:04:36.739049 2547 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 14:04:37.237658 2325 raft_consensus.cc:491] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0c8941c4e30348ed9bc4a3262cd9fe99)
I20250626 14:04:37.237815 2325 raft_consensus.cc:513] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:37.238252 2325 leader_election.cc:290] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:37.238701 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b325108f7de949199cc10dbe259a0651" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
W20250626 14:04:37.239024 1748 leader_election.cc:343] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: b325108f7de949199cc10dbe259a0651
I20250626 14:04:37.239128 1749 leader_election.cc:304] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e517893e13f546c0b40d976633b49651; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b
I20250626 14:04:37.239313 2325 raft_consensus.cc:2747] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 14:04:37.284080 2553 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 07532b7648e4414789ed2b099bad2b19)
I20250626 14:04:37.284210 2553 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:37.284718 2553 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:04:37.285843 2419 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:04:37.285936 2417 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:04:37.286028 2417 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:04:37.286164 2553 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 14:04:37.354689 2557 raft_consensus.cc:491] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0c8941c4e30348ed9bc4a3262cd9fe99)
I20250626 14:04:37.354869 2557 raft_consensus.cc:513] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:37.355417 2557 leader_election.cc:290] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:37.355882 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b325108f7de949199cc10dbe259a0651" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:37.356030 1813 raft_consensus.cc:2466] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
W20250626 14:04:37.356233 1613 leader_election.cc:343] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: b325108f7de949199cc10dbe259a0651
I20250626 14:04:37.356410 1613 leader_election.cc:304] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99
I20250626 14:04:37.356671 2557 raft_consensus.cc:2802] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 14:04:37.356743 2557 raft_consensus.cc:491] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0c8941c4e30348ed9bc4a3262cd9fe99)
I20250626 14:04:37.356809 2557 raft_consensus.cc:3058] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Advancing to term 2
I20250626 14:04:37.357775 2557 raft_consensus.cc:513] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:37.358148 2557 leader_election.cc:290] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 election: Requested vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:37.358579 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b325108f7de949199cc10dbe259a0651" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:37.358728 1813 raft_consensus.cc:3058] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Advancing to term 2
W20250626 14:04:37.358935 1613 leader_election.cc:343] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: b325108f7de949199cc10dbe259a0651
I20250626 14:04:37.359625 1813 raft_consensus.cc:2466] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 2.
I20250626 14:04:37.360317 1613 leader_election.cc:304] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99
I20250626 14:04:37.360560 2557 raft_consensus.cc:2802] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [term 2 FOLLOWER]: Leader election won for term 2
I20250626 14:04:37.360652 2557 raft_consensus.cc:695] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [term 2 LEADER]: Becoming Leader. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Running, Role: LEADER
I20250626 14:04:37.360802 2557 consensus_queue.cc:237] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } }
I20250626 14:04:37.793202 1813 raft_consensus.cc:1273] T b325108f7de949199cc10dbe259a0651 P e517893e13f546c0b40d976633b49651 [term 2 FOLLOWER]: Refusing update from remote peer 4e57adb83e9f451488ed1d0034043e7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 14:04:37.793670 2557 consensus_queue.cc:1035] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 14:04:37.798199 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 1: this message will repeat every 5th retry.
I20250626 14:04:38.024880 2570 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 07532b7648e4414789ed2b099bad2b19)
I20250626 14:04:38.025022 2570 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:38.025540 2570 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:04:38.026736 2417 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:04:38.026839 2419 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:38.026929 2419 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:04:38.027081 2570 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 14:04:38.452245 1071 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmpJjah/build/debug/bin/kudu with pid 2399
I20250626 14:04:39.204775 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.254:36707
--webserver_interface=127.1.11.254
--webserver_port=45241
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250626 14:04:39.315274 2585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:39.315507 2585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:39.315548 2585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:04:39.318732 2585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:04:39.318785 2585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:39.318806 2585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:04:39.318825 2585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:04:39.322817 2585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.254:36707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.11.254
--webserver_port=45241
--never_fsync=true
--heap_profile_path=/tmp/kudu.2585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:39.323849 2585 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:39.324949 2585 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:04:39.331604 2594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:39.331597 2591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:39.331768 2592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:39.331974 2585 server_base.cc:1048] running on GCE node
I20250626 14:04:39.332482 2585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:39.333388 2585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:39.334543 2585 hybrid_clock.cc:648] HybridClock initialized: now 1750946679334535 us; error 29 us; skew 500 ppm
I20250626 14:04:39.336410 2585 webserver.cc:469] Webserver started at http://127.1.11.254:45241/ using document root <none> and password file <none>
I20250626 14:04:39.336987 2585 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:39.337044 2585 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:39.340413 2585 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20250626 14:04:39.342303 2600 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:39.343302 2585 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:39.343401 2585 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
uuid: "24ad911d71b6433f8c7939a1d1f7b782"
format_stamp: "Formatted at 2025-06-26 14:04:30 on dist-test-slave-rptj"
I20250626 14:04:39.343801 2585 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:04:39.367146 2585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:39.367805 2585 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:39.367954 2585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:39.374811 2585 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.254:36707
I20250626 14:04:39.374819 2652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.254:36707 every 8 connection(s)
I20250626 14:04:39.375943 2585 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/info.pb
I20250626 14:04:39.379571 2653 sys_catalog.cc:263] Verifying existing consensus state
I20250626 14:04:39.381218 2653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Bootstrap starting.
I20250626 14:04:39.381306 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 2585
I20250626 14:04:39.391068 2653 log.cc:826] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Log is configured to *not* fsync() on all Append() calls
I20250626 14:04:39.426321 2653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Bootstrap replayed 1/1 log segments. Stats: ops{read=71 overwritten=0 applied=71 ignored=0} inserts{seen=49 ignored=0} mutations{seen=58 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 14:04:39.426743 2653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Bootstrap complete.
I20250626 14:04:39.429368 2653 raft_consensus.cc:357] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:39.429744 2653 raft_consensus.cc:738] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ad911d71b6433f8c7939a1d1f7b782, State: Initialized, Role: FOLLOWER
I20250626 14:04:39.430160 2653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 4.71, Last appended by leader: 71, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:39.430611 2657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:39.430745 2657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:39.431231 2653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:04:39.444832 2671 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:39.444968 2671 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: loading cluster ID for follower catalog manager: success
I20250626 14:04:39.444732 2585 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:04:39.446784 2585 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:04:39.448982 2671 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: acquiring CA information for follower catalog manager: success
I20250626 14:04:39.451054 2671 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 14:04:39.453050 1725 heartbeater.cc:344] Connected to a master server at 127.1.11.254:36707
I20250626 14:04:39.454934 2617 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" instance_seqno: 1750946671637378) as {username='slave'} at 127.1.11.193:55761; Asking this server to re-register.
I20250626 14:04:39.455439 1725 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:39.456617 2617 ts_manager.cc:194] Registered new tserver with Master: 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:39.719446 1860 heartbeater.cc:344] Connected to a master server at 127.1.11.254:36707
I20250626 14:04:39.720671 2617 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e517893e13f546c0b40d976633b49651" instance_seqno: 1750946671821196) as {username='slave'} at 127.1.11.194:42445; Asking this server to re-register.
I20250626 14:04:39.721103 1860 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:39.722401 2617 ts_manager.cc:194] Registered new tserver with Master: e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:39.742220 1995 heartbeater.cc:344] Connected to a master server at 127.1.11.254:36707
I20250626 14:04:39.743084 2617 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" instance_seqno: 1750946672002466) as {username='slave'} at 127.1.11.195:54239; Asking this server to re-register.
I20250626 14:04:39.743552 1995 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:39.744359 2617 ts_manager.cc:194] Registered new tserver with Master: 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:39.858824 2657 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:39.859046 2657 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:39.860296 2657 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:04:39.860842 2603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:04:39.861685 2603 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:04:39.861799 2601 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:04:39.861892 2601 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:04:39.862100 2657 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 14:04:40.530611 2681 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:40.530769 2681 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:40.531208 2681 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:04:40.532351 2601 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:04:40.532514 2603 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:40.532613 2603 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:04:40.532757 2681 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 14:04:41.048242 2687 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:41.048370 2687 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:41.048861 2687 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:04:41.050117 2601 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:04:41.050249 2603 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:41.050294 2603 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:04:41.050406 2687 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250626 14:04:41.087083 1748 consensus_peers.cc:487] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3. This is attempt 6: this message will repeat every 5th retry.
W20250626 14:04:41.461766 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 6: this message will repeat every 5th retry.
W20250626 14:04:41.559626 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 6: this message will repeat every 5th retry.
W20250626 14:04:41.686467 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 6: this message will repeat every 5th retry.
I20250626 14:04:42.462386 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.252:37925
--webserver_interface=127.1.11.252
--webserver_port=43659
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250626 14:04:42.524700 2698 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:42.524837 2698 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:42.525327 2698 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:04:42.526708 2601 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:04:42.526829 2603 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:42.526939 2603 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:04:42.527086 2698 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250626 14:04:42.570403 2697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:42.570631 2697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:42.570703 2697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:04:42.574222 2697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:04:42.574290 2697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:42.574311 2697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:04:42.574331 2697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:04:42.578214 2697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.252:37925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.11.252
--webserver_port=43659
--never_fsync=true
--heap_profile_path=/tmp/kudu.2697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:42.579315 2697 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:42.580493 2697 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:04:42.587013 2707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:42.587077 2706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:42.587157 2697 server_base.cc:1048] running on GCE node
W20250626 14:04:42.587018 2709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:42.587746 2697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:42.588662 2697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:42.589820 2697 hybrid_clock.cc:648] HybridClock initialized: now 1750946682589814 us; error 30 us; skew 500 ppm
I20250626 14:04:42.597416 2697 webserver.cc:469] Webserver started at http://127.1.11.252:43659/ using document root <none> and password file <none>
I20250626 14:04:42.598047 2697 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:42.598127 2697 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:42.601450 2697 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20250626 14:04:42.603487 2715 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:42.604768 2697 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250626 14:04:42.604861 2697 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
uuid: "81db61eed5584e7e8b0cf40e5441f88e"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:42.605216 2697 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:04:42.621390 2697 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:42.622046 2697 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:42.622198 2697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:42.629468 2767 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.252:37925 every 8 connection(s)
I20250626 14:04:42.629447 2697 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.252:37925
I20250626 14:04:42.630609 2697 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/info.pb
I20250626 14:04:42.634464 2768 sys_catalog.cc:263] Verifying existing consensus state
I20250626 14:04:42.636031 2768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Bootstrap starting.
I20250626 14:04:42.638724 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 2697
W20250626 14:04:42.642614 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 6: this message will repeat every 5th retry.
I20250626 14:04:42.647473 2768 log.cc:826] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Log is configured to *not* fsync() on all Append() calls
I20250626 14:04:42.649277 1724 heartbeater.cc:344] Connected to a master server at 127.1.11.252:37925
I20250626 14:04:42.681074 1859 heartbeater.cc:344] Connected to a master server at 127.1.11.252:37925
I20250626 14:04:42.683030 2768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Bootstrap replayed 1/1 log segments. Stats: ops{read=63 overwritten=0 applied=63 ignored=0} inserts{seen=44 ignored=0} mutations{seen=51 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 14:04:42.683393 2768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Bootstrap complete.
I20250626 14:04:42.685994 2768 raft_consensus.cc:357] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:42.686316 2768 raft_consensus.cc:738] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81db61eed5584e7e8b0cf40e5441f88e, State: Initialized, Role: FOLLOWER
I20250626 14:04:42.686767 2768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 63, Last appended: 3.63, Last appended by leader: 63, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:42.687414 2773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:42.687525 2773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:42.687836 2768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:04:42.693760 2787 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:42.693890 2787 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: loading cluster ID for follower catalog manager: success
I20250626 14:04:42.696477 2787 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: acquiring CA information for follower catalog manager: success
I20250626 14:04:42.698366 2787 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 14:04:42.704646 2697 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:04:42.706516 2697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:04:42.739141 2731 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e517893e13f546c0b40d976633b49651" instance_seqno: 1750946671821196) as {username='slave'} at 127.1.11.194:54913; Asking this server to re-register.
I20250626 14:04:42.739737 1859 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:42.740980 2731 ts_manager.cc:194] Registered new tserver with Master: e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:42.747805 1994 heartbeater.cc:344] Connected to a master server at 127.1.11.252:37925
I20250626 14:04:42.748999 2731 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" instance_seqno: 1750946672002466) as {username='slave'} at 127.1.11.195:41445; Asking this server to re-register.
I20250626 14:04:42.749444 1994 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:42.750279 2731 ts_manager.cc:194] Registered new tserver with Master: 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:43.024564 2731 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" instance_seqno: 1750946671637378) as {username='slave'} at 127.1.11.193:46781; Asking this server to re-register.
I20250626 14:04:43.025103 1724 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:43.026124 2731 ts_manager.cc:194] Registered new tserver with Master: 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:43.143915 2773 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:43.144146 2773 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:43.145467 2773 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
W20250626 14:04:43.146045 2718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:43.145999 2627 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81db61eed5584e7e8b0cf40e5441f88e" candidate_term: 4 candidate_status { last_received { term: 3 index: 63 } } ignore_live_leader: false dest_uuid: "24ad911d71b6433f8c7939a1d1f7b782" is_pre_election: true
I20250626 14:04:43.146327 2627 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 81db61eed5584e7e8b0cf40e5441f88e in current term 4: Already voted for candidate 07532b7648e4414789ed2b099bad2b19 in this term.
W20250626 14:04:43.147157 2718 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:43.147244 2718 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:43.147490 2773 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 3 FOLLOWER]: Advancing to term 4
I20250626 14:04:43.149420 2773 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 14:04:43.789032 2797 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:43.789165 2797 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:43.789579 2797 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
I20250626 14:04:43.790037 2627 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81db61eed5584e7e8b0cf40e5441f88e" candidate_term: 5 candidate_status { last_received { term: 3 index: 63 } } ignore_live_leader: false dest_uuid: "24ad911d71b6433f8c7939a1d1f7b782" is_pre_election: true
I20250626 14:04:43.790246 2627 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 81db61eed5584e7e8b0cf40e5441f88e for term 5 because replica has last-logged OpId of term: 4 index: 71, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 63.
W20250626 14:04:43.790855 2718 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:43.790925 2718 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:43.791049 2797 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 14:04:44.448900 2585 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:04:44.449019 2585 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:04:44.698455 2803 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:44.698588 2803 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:44.699048 2803 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
I20250626 14:04:44.699456 2627 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81db61eed5584e7e8b0cf40e5441f88e" candidate_term: 5 candidate_status { last_received { term: 3 index: 63 } } ignore_live_leader: false dest_uuid: "24ad911d71b6433f8c7939a1d1f7b782" is_pre_election: true
I20250626 14:04:44.699631 2627 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 81db61eed5584e7e8b0cf40e5441f88e for term 5 because replica has last-logged OpId of term: 4 index: 71, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 63.
W20250626 14:04:44.700289 2718 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:44.700397 2718 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:44.700521 2803 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 14:04:44.831329 1071 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmpJjah/build/debug/bin/kudu with pid 2697
I20250626 14:04:44.908495 2808 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:44.908713 2808 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:44.909211 2808 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:04:44.909672 2603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20250626 14:04:44.910377 2601 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:04:44.910472 2603 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:44.910552 2603 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:04:44.910704 2808 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 14:04:45.435568 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.252:37925
--webserver_interface=127.1.11.252
--webserver_port=43659
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250626 14:04:45.546581 2815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:45.546831 2815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:45.546900 2815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:04:45.550253 2815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:04:45.550339 2815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:45.550374 2815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:04:45.550406 2815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:04:45.554319 2815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.252:37925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.11.252
--webserver_port=43659
--never_fsync=true
--heap_profile_path=/tmp/kudu.2815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:45.555542 2815 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:45.556823 2815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:04:45.563625 2820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:45.563666 2823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:45.563635 2821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:45.563850 2815 server_base.cc:1048] running on GCE node
I20250626 14:04:45.564400 2815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:45.565413 2815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:45.566566 2815 hybrid_clock.cc:648] HybridClock initialized: now 1750946685566558 us; error 30 us; skew 500 ppm
I20250626 14:04:45.568600 2815 webserver.cc:469] Webserver started at http://127.1.11.252:43659/ using document root <none> and password file <none>
I20250626 14:04:45.569226 2815 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:45.569293 2815 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:45.572727 2815 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20250626 14:04:45.574712 2829 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:45.575760 2815 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:45.575874 2815 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
uuid: "81db61eed5584e7e8b0cf40e5441f88e"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:45.576264 2815 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:04:45.595909 2815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:45.596892 2815 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:45.597075 2815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:45.604573 2815 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.252:37925
I20250626 14:04:45.604596 2881 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.252:37925 every 8 connection(s)
I20250626 14:04:45.605895 2815 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/info.pb
I20250626 14:04:45.609889 2882 sys_catalog.cc:263] Verifying existing consensus state
I20250626 14:04:45.611542 2882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Bootstrap starting.
I20250626 14:04:45.611954 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 2815
I20250626 14:04:45.622886 2882 log.cc:826] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Log is configured to *not* fsync() on all Append() calls
I20250626 14:04:45.654788 2882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Bootstrap replayed 1/1 log segments. Stats: ops{read=63 overwritten=0 applied=63 ignored=0} inserts{seen=44 ignored=0} mutations{seen=51 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 14:04:45.655347 2882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Bootstrap complete.
I20250626 14:04:45.658260 2882 raft_consensus.cc:357] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:45.658640 2882 raft_consensus.cc:738] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81db61eed5584e7e8b0cf40e5441f88e, State: Initialized, Role: FOLLOWER
I20250626 14:04:45.659080 2882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 63, Last appended: 3.63, Last appended by leader: 63, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:45.659533 2886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:45.659664 2886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:45.660241 2882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:04:45.666873 2900 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:45.667011 2900 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: loading cluster ID for follower catalog manager: success
I20250626 14:04:45.669879 2900 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: acquiring CA information for follower catalog manager: success
I20250626 14:04:45.671830 2900 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 14:04:45.672509 2815 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:04:45.674269 2815 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:04:45.763599 1994 heartbeater.cc:344] Connected to a master server at 127.1.11.252:37925
I20250626 14:04:45.765597 2846 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" instance_seqno: 1750946672002466) as {username='slave'} at 127.1.11.195:59627; Asking this server to re-register.
I20250626 14:04:45.766151 1994 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:45.767457 2846 ts_manager.cc:194] Registered new tserver with Master: 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:45.783016 1859 heartbeater.cc:344] Connected to a master server at 127.1.11.252:37925
I20250626 14:04:45.784454 2846 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e517893e13f546c0b40d976633b49651" instance_seqno: 1750946671821196) as {username='slave'} at 127.1.11.194:45595; Asking this server to re-register.
I20250626 14:04:45.784965 1859 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:45.785984 2846 ts_manager.cc:194] Registered new tserver with Master: e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
W20250626 14:04:45.865011 1748 consensus_peers.cc:487] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3. This is attempt 11: this message will repeat every 5th retry.
I20250626 14:04:45.938469 1724 heartbeater.cc:344] Connected to a master server at 127.1.11.252:37925
I20250626 14:04:45.939478 2846 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" instance_seqno: 1750946671637378) as {username='slave'} at 127.1.11.193:45169; Asking this server to re-register.
I20250626 14:04:45.939877 1724 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:45.940717 2846 ts_manager.cc:194] Registered new tserver with Master: 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:46.200467 2907 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:46.200753 2907 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:46.202018 2907 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
I20250626 14:04:46.202455 2627 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81db61eed5584e7e8b0cf40e5441f88e" candidate_term: 5 candidate_status { last_received { term: 3 index: 63 } } ignore_live_leader: false dest_uuid: "24ad911d71b6433f8c7939a1d1f7b782" is_pre_election: true
I20250626 14:04:46.202630 2627 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 81db61eed5584e7e8b0cf40e5441f88e for term 5 because replica has last-logged OpId of term: 4 index: 71, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 63.
W20250626 14:04:46.202646 2832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:04:46.203511 2832 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:46.203611 2832 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:46.203842 2907 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250626 14:04:46.708262 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 11: this message will repeat every 5th retry.
W20250626 14:04:46.770784 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 11: this message will repeat every 5th retry.
W20250626 14:04:46.881528 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 11: this message will repeat every 5th retry.
I20250626 14:04:46.909229 2911 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:46.909371 2911 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:46.909818 2911 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
I20250626 14:04:46.910270 2627 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81db61eed5584e7e8b0cf40e5441f88e" candidate_term: 5 candidate_status { last_received { term: 3 index: 63 } } ignore_live_leader: false dest_uuid: "24ad911d71b6433f8c7939a1d1f7b782" is_pre_election: true
I20250626 14:04:46.910461 2627 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 81db61eed5584e7e8b0cf40e5441f88e for term 5 because replica has last-logged OpId of term: 4 index: 71, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 63.
W20250626 14:04:46.911056 2832 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:46.911121 2832 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:46.911242 2911 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 14:04:47.022997 2914 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:47.023126 2914 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:47.023970 2914 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:04:47.025081 2603 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:47.028348 2856 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24ad911d71b6433f8c7939a1d1f7b782" candidate_term: 5 candidate_status { last_received { term: 4 index: 71 } } ignore_live_leader: false dest_uuid: "81db61eed5584e7e8b0cf40e5441f88e" is_pre_election: true
I20250626 14:04:47.028631 2856 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ad911d71b6433f8c7939a1d1f7b782 in term 4.
I20250626 14:04:47.028978 2601 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782, 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19
I20250626 14:04:47.029148 2914 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 14:04:47.029217 2914 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:47.029248 2914 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 4 FOLLOWER]: Advancing to term 5
I20250626 14:04:47.030969 2914 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:47.031339 2914 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 election: Requested vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
I20250626 14:04:47.031774 2856 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24ad911d71b6433f8c7939a1d1f7b782" candidate_term: 5 candidate_status { last_received { term: 4 index: 71 } } ignore_live_leader: false dest_uuid: "81db61eed5584e7e8b0cf40e5441f88e"
I20250626 14:04:47.031914 2856 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 4 FOLLOWER]: Advancing to term 5
W20250626 14:04:47.032512 2603 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:47.033828 2856 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ad911d71b6433f8c7939a1d1f7b782 in term 5.
I20250626 14:04:47.034121 2601 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782, 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19
I20250626 14:04:47.034286 2914 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 14:04:47.034535 2914 raft_consensus.cc:695] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 5 LEADER]: Becoming Leader. State: Replica: 24ad911d71b6433f8c7939a1d1f7b782, State: Running, Role: LEADER
I20250626 14:04:47.034880 2914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 4.71, Last appended by leader: 71, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:47.036607 2919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24ad911d71b6433f8c7939a1d1f7b782. Latest consensus state: current_term: 5 leader_uuid: "24ad911d71b6433f8c7939a1d1f7b782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:47.036741 2919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:47.037061 2921 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 14:04:47.038694 2921 catalog_manager.cc:671] Loaded metadata for table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]
I20250626 14:04:47.038962 2921 catalog_manager.cc:671] Loaded metadata for table default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843]
I20250626 14:04:47.039167 2921 catalog_manager.cc:671] Loaded metadata for table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]
I20250626 14:04:47.039471 2921 catalog_manager.cc:671] Loaded metadata for table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]
I20250626 14:04:47.039682 2921 catalog_manager.cc:671] Loaded metadata for table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9]
I20250626 14:04:47.039911 2921 catalog_manager.cc:671] Loaded metadata for table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]
I20250626 14:04:47.040109 2921 catalog_manager.cc:671] Loaded metadata for table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]
I20250626 14:04:47.040313 2921 catalog_manager.cc:671] Loaded metadata for table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]
I20250626 14:04:47.040515 2921 catalog_manager.cc:671] Loaded metadata for table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]
I20250626 14:04:47.040773 2921 catalog_manager.cc:671] Loaded metadata for table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]
I20250626 14:04:47.041051 2921 catalog_manager.cc:671] Loaded metadata for table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]
I20250626 14:04:47.042102 2921 tablet_loader.cc:96] loaded metadata for tablet 025415f7a05746e19d5ed199246fc949 (table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a])
I20250626 14:04:47.042429 2921 tablet_loader.cc:96] loaded metadata for tablet 0db0b418e8b8412da3dc098ef6aab940 (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140])
I20250626 14:04:47.042603 2921 tablet_loader.cc:96] loaded metadata for tablet 17a5b09fa0b54d45b835aa64dac8a1af (table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447])
I20250626 14:04:47.042762 2921 tablet_loader.cc:96] loaded metadata for tablet 1e307e729d7d42a68807263959c223a6 (table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb])
I20250626 14:04:47.042908 2921 tablet_loader.cc:96] loaded metadata for tablet 33d002d179974a95aaad3d84814d27f0 (table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75])
I20250626 14:04:47.043159 2921 tablet_loader.cc:96] loaded metadata for tablet 5f5ce7fc87164ae1ace91557275285f8 (table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233])
I20250626 14:04:47.043743 2921 tablet_loader.cc:96] loaded metadata for tablet 730ed7238006463eb08803a70330a43f (table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9])
I20250626 14:04:47.043905 2921 tablet_loader.cc:96] loaded metadata for tablet 7c18e8a737f548c6ab77f739ca17135b (table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80])
I20250626 14:04:47.044302 2921 tablet_loader.cc:96] loaded metadata for tablet b25221169c9c48718276736fe657f378 (table default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843])
I20250626 14:04:47.044869 2921 tablet_loader.cc:96] loaded metadata for tablet dd5f0eb6de154b3c9ffdd6d8f7042a49 (table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03])
I20250626 14:04:47.045018 2921 tablet_loader.cc:96] loaded metadata for tablet f1c96426790242fdb8224137d55939f2 (table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c])
I20250626 14:04:47.045387 2921 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 14:04:47.045974 2921 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:47.046021 2921 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 14:04:47.046756 2921 catalog_manager.cc:1506] Loading token signing keys...
I20250626 14:04:47.047295 2921 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Loaded TSK: 0
I20250626 14:04:47.048076 2921 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 14:04:47.048703 2921 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250626 14:04:47.049188 2921 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 13
I20250626 14:04:47.071070 2616 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44220:
name: "default.Table_213d95d0e97546c298d94c45091826cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:47.071245 2614 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44220:
name: "default.Table_fdd6a46b24984290ac6f503f5684b003"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:47.071555 2609 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44220:
name: "default.Table_f7012cd4bf5d40d6af93cf9b01405dcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:47.071082 2617 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44220:
name: "default.Table_8c238e2636e9491f938a48294f961982"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:47.071758 2612 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44220:
table { table_name: "default.Table_22f5000aea364833a97948c4478f7e27" } new_table_name: "default.Table_8268b369fb20436fb4e562f6a97eee1d" modify_external_catalogs: true
I20250626 14:04:47.072080 2610 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44220:
table { table_name: "default.Table_0b500f44e16e45e589c01ab7ad9a621b" } modify_external_catalogs: true
I20250626 14:04:47.072216 2610 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44220:
table { table_name: "default.Table_0b500f44e16e45e589c01ab7ad9a621b" } modify_external_catalogs: true
I20250626 14:04:47.072306 2613 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44220:
name: "default.Table_566f55c5d8c34043a37041fef8cb9f45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:47.072213 2615 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44220:
name: "default.Table_2c8dee0163784a60a8468b53619247f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:47.072471 2608 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44220:
table { table_name: "default.Table_3a3f21b3c0e54e438561e02ab79fa36b" } new_table_name: "default.Table_3633fc38c5374180a081fab63a1eed84" modify_external_catalogs: true
I20250626 14:04:47.072757 2611 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44220:
name: "default.Table_33ee3521f64d4cefb8856259510e5f0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250626 14:04:47.076143 2670 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 16, previous 16; dumping first 4 out of 8 received events: NotificationEvent(eventId=14, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_72f9307d2f5247a0ae4499c953ea1650, message={"server":"","servicePrincipal":"","db":"default","table":"table_72f9307d2f5247a0ae4499c953ea1650","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_72f9307d2f5247a0ae4499c953ea1650\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_72f9307d2f5247a0ae4499c953ea1650\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_72f9307d2f5247a0ae4499c953ea1650\",\"kudu.table_id\":\"61be84e63b2e478b9da5619fdbdffee9\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=15, eventTime=1750946674, eventType=CREATE_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=ALTER_TABLE, dbName=default, tableName=table_7d5a9b0c3ae440b89dac3ca5dd8e81ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_fb8ebc7aa8894f28b4d79252382f356a","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_fb8ebc7aa8894f28b4d79252382f356a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_fb8ebc7aa8894f28b4d79252382f356a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fb8ebc7aa8894f28b4d79252382f356a\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7eff8db88dcd google::LogMessage::Fail() at ??:0
@ 0x7eff8db8cb93 google::LogMessage::SendToLog() at ??:0
@ 0x7eff8db887cc google::LogMessage::Flush() at ??:0
@ 0x7eff8db88a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7eff92b21c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7eff92b20ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7eff92b2030b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7eff92b25a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56092ba72752 std::function<>::operator()() at ??:0
@ 0x7eff8efda0a1 kudu::Thread::SuperviseThread() at ??:0
@ 0x7eff8f7c96db start_thread at ??:0
@ 0x7eff8cd3f71f clone at ??:0
W20250626 14:04:47.263489 2001 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.11.254:36707)
W20250626 14:04:47.263995 2001 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.11.254:36707)
W20250626 14:04:47.264139 2001 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.11.254:36707)
W20250626 14:04:47.264261 2001 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.11.254:36707)
W20250626 14:04:47.266441 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 11: this message will repeat every 5th retry.
I20250626 14:04:47.832640 1071 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmpJjah/build/debug/bin/kudu with pid 2815
W20250626 14:04:47.839440 2001 connection.cc:537] client connection to 127.1.11.252:37925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250626 14:04:48.439953 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.252:37925
--webserver_interface=127.1.11.252
--webserver_port=43659
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250626 14:04:48.547735 2948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:48.548027 2948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:48.548100 2948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:04:48.551193 2948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:04:48.551260 2948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:48.551293 2948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:04:48.551324 2948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:04:48.555475 2948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.252:37925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.11.252
--webserver_port=43659
--never_fsync=true
--heap_profile_path=/tmp/kudu.2948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:48.556643 2948 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:48.557798 2948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:04:48.564669 2955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:48.564661 2957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:48.564662 2954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:48.564755 2948 server_base.cc:1048] running on GCE node
I20250626 14:04:48.565498 2948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:48.566429 2948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:48.567584 2948 hybrid_clock.cc:648] HybridClock initialized: now 1750946688567567 us; error 36 us; skew 500 ppm
I20250626 14:04:48.574024 2948 webserver.cc:469] Webserver started at http://127.1.11.252:43659/ using document root <none> and password file <none>
I20250626 14:04:48.574683 2948 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:48.574756 2948 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:48.578434 2948 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20250626 14:04:48.580405 2963 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:48.581467 2948 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:48.581578 2948 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
uuid: "81db61eed5584e7e8b0cf40e5441f88e"
format_stamp: "Formatted at 2025-06-26 14:04:31 on dist-test-slave-rptj"
I20250626 14:04:48.581961 2948 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:04:48.619365 2948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:48.620055 2948 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:48.620230 2948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:48.627156 2948 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.252:37925
I20250626 14:04:48.627175 3015 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.252:37925 every 8 connection(s)
I20250626 14:04:48.628253 2948 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-2/data/info.pb
I20250626 14:04:48.631991 3016 sys_catalog.cc:263] Verifying existing consensus state
I20250626 14:04:48.633620 3016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Bootstrap starting.
I20250626 14:04:48.636449 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 2948
I20250626 14:04:48.643204 3016 log.cc:826] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Log is configured to *not* fsync() on all Append() calls
I20250626 14:04:48.663654 1859 heartbeater.cc:344] Connected to a master server at 127.1.11.252:37925
I20250626 14:04:48.679124 3016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Bootstrap replayed 1/1 log segments. Stats: ops{read=63 overwritten=0 applied=63 ignored=0} inserts{seen=44 ignored=0} mutations{seen=51 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 14:04:48.679556 3016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Bootstrap complete.
I20250626 14:04:48.682427 3016 raft_consensus.cc:357] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:48.682777 3016 raft_consensus.cc:738] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81db61eed5584e7e8b0cf40e5441f88e, State: Initialized, Role: FOLLOWER
I20250626 14:04:48.683211 3016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 63, Last appended: 3.63, Last appended by leader: 63, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:48.683673 3021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:48.683784 3021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:48.684283 3016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:04:48.690443 3035 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:48.690506 3035 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: loading cluster ID for follower catalog manager: success
I20250626 14:04:48.692447 3035 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: acquiring CA information for follower catalog manager: success
I20250626 14:04:48.693854 3035 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 14:04:48.694670 2948 master_runner.cc:186] Error getting master registration for 127.1.11.254:36707: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111),
I20250626 14:04:48.696879 2948 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:04:48.742216 2980 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e517893e13f546c0b40d976633b49651" instance_seqno: 1750946671821196) as {username='slave'} at 127.1.11.194:42985; Asking this server to re-register.
I20250626 14:04:48.742802 1859 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:48.744158 2980 ts_manager.cc:194] Registered new tserver with Master: e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:48.779893 1994 heartbeater.cc:344] Connected to a master server at 127.1.11.252:37925
I20250626 14:04:48.780897 2980 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" instance_seqno: 1750946672002466) as {username='slave'} at 127.1.11.195:47527; Asking this server to re-register.
I20250626 14:04:48.781304 1994 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:48.782202 2980 ts_manager.cc:194] Registered new tserver with Master: 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:48.795671 1724 heartbeater.cc:344] Connected to a master server at 127.1.11.252:37925
I20250626 14:04:48.797046 2980 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" instance_seqno: 1750946671637378) as {username='slave'} at 127.1.11.193:34319; Asking this server to re-register.
I20250626 14:04:48.797580 1724 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:48.798676 2980 ts_manager.cc:194] Registered new tserver with Master: 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:49.273950 3040 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:49.274186 3040 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:49.275454 3040 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
W20250626 14:04:49.275997 2964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
W20250626 14:04:49.276729 2966 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:04:49.276842 2964 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
I20250626 14:04:49.276929 2964 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:49.277177 3040 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 14:04:49.888701 3044 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:49.888837 3044 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:49.889281 3044 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
W20250626 14:04:49.890425 2964 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
W20250626 14:04:49.890511 2966 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:49.890614 2966 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:49.890758 3044 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250626 14:04:50.914044 1748 consensus_peers.cc:487] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3. This is attempt 16: this message will repeat every 5th retry.
I20250626 14:04:50.947033 3050 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:50.947185 3050 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:50.947636 3050 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
W20250626 14:04:50.949040 2964 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
W20250626 14:04:50.949136 2966 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:50.949198 2966 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:50.949352 3050 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250626 14:04:51.446622 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 16: this message will repeat every 5th retry.
W20250626 14:04:51.661453 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 16: this message will repeat every 5th retry.
W20250626 14:04:51.680142 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 16: this message will repeat every 5th retry.
I20250626 14:04:51.858778 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.253:44447
--webserver_interface=127.1.11.253
--webserver_port=37937
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250626 14:04:51.967110 3056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:51.967407 3056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:51.967483 3056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:04:51.970842 3056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:04:51.970906 3056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:51.970927 3056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:04:51.970944 3056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:04:51.974790 3056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.253:44447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.11.253
--webserver_port=37937
--never_fsync=true
--heap_profile_path=/tmp/kudu.3056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:51.975916 3056 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:51.977080 3056 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:04:51.983947 3061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:51.983952 3062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:51.983949 3064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:51.984433 3056 server_base.cc:1048] running on GCE node
I20250626 14:04:51.984993 3056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:51.985932 3056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:51.987201 3056 hybrid_clock.cc:648] HybridClock initialized: now 1750946691987187 us; error 24 us; skew 500 ppm
I20250626 14:04:51.989125 3056 webserver.cc:469] Webserver started at http://127.1.11.253:37937/ using document root <none> and password file <none>
I20250626 14:04:51.989719 3056 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:51.989782 3056 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:51.993302 3056 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250626 14:04:51.995242 3070 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:51.996398 3056 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:51.996516 3056 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
uuid: "07532b7648e4414789ed2b099bad2b19"
format_stamp: "Formatted at 2025-06-26 14:04:30 on dist-test-slave-rptj"
I20250626 14:04:51.996932 3056 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:04:52.013705 3056 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:52.014329 3056 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:52.014501 3056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:52.021466 3056 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.253:44447
I20250626 14:04:52.021482 3122 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.253:44447 every 8 connection(s)
I20250626 14:04:52.022604 3056 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
I20250626 14:04:52.025490 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 3056
I20250626 14:04:52.026810 3123 sys_catalog.cc:263] Verifying existing consensus state
I20250626 14:04:52.028468 3123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Bootstrap starting.
I20250626 14:04:52.037038 3123 log.cc:826] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Log is configured to *not* fsync() on all Append() calls
I20250626 14:04:52.070585 3123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Bootstrap replayed 1/1 log segments. Stats: ops{read=71 overwritten=0 applied=71 ignored=0} inserts{seen=49 ignored=0} mutations{seen=58 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 14:04:52.071022 3123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Bootstrap complete.
I20250626 14:04:52.073789 3123 raft_consensus.cc:357] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:52.074146 3123 raft_consensus.cc:738] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07532b7648e4414789ed2b099bad2b19, State: Initialized, Role: FOLLOWER
I20250626 14:04:52.074622 3123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 4.71, Last appended by leader: 71, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:52.075122 3127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:52.075289 3127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:52.075721 3123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:04:52.082439 3141 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:52.082578 3141 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: loading cluster ID for follower catalog manager: success
I20250626 14:04:52.081043 3056 master_runner.cc:186] Error getting master registration for 127.1.11.254:36707: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111),
I20250626 14:04:52.083230 3056 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:04:52.084959 3141 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: acquiring CA information for follower catalog manager: success
I20250626 14:04:52.088351 3141 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 14:04:52.105496 1861 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:52.107908 3084 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e517893e13f546c0b40d976633b49651" instance_seqno: 1750946671821196) as {username='slave'} at 127.1.11.194:43641; Asking this server to re-register.
I20250626 14:04:52.108609 1861 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:52.110018 3084 ts_manager.cc:194] Registered new tserver with Master: e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:52.244470 1726 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:52.244643 1726 heartbeater.cc:499] Master 127.1.11.253:44447 was elected leader, sending a full tablet report...
I20250626 14:04:52.245815 3084 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" instance_seqno: 1750946671637378) as {username='slave'} at 127.1.11.193:43285; Asking this server to re-register.
I20250626 14:04:52.246212 1726 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:52.247105 3084 ts_manager.cc:194] Registered new tserver with Master: 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:52.267391 1996 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:52.268445 3084 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" instance_seqno: 1750946672002466) as {username='slave'} at 127.1.11.195:34521; Asking this server to re-register.
I20250626 14:04:52.268922 1996 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:52.270220 3084 ts_manager.cc:194] Registered new tserver with Master: 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:04:52.304883 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 16: this message will repeat every 5th retry.
I20250626 14:04:52.440199 3145 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:52.440353 3145 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:52.440841 3145 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
W20250626 14:04:52.442281 2964 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
I20250626 14:04:52.444727 3097 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81db61eed5584e7e8b0cf40e5441f88e" candidate_term: 6 candidate_status { last_received { term: 3 index: 63 } } ignore_live_leader: false dest_uuid: "07532b7648e4414789ed2b099bad2b19" is_pre_election: true
I20250626 14:04:52.445051 3097 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 81db61eed5584e7e8b0cf40e5441f88e for term 6 because replica has last-logged OpId of term: 4 index: 71, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 63.
I20250626 14:04:52.445608 2966 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:52.445778 3145 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 14:04:52.566077 3127 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:52.566318 3127 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:52.567516 3127 leader_election.cc:290] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:04:52.568252 3071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
W20250626 14:04:52.569255 3071 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
I20250626 14:04:52.571626 2990 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07532b7648e4414789ed2b099bad2b19" candidate_term: 5 candidate_status { last_received { term: 4 index: 71 } } ignore_live_leader: false dest_uuid: "81db61eed5584e7e8b0cf40e5441f88e" is_pre_election: true
I20250626 14:04:52.571924 2990 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 07532b7648e4414789ed2b099bad2b19 in current term 5: Already voted for candidate 24ad911d71b6433f8c7939a1d1f7b782 in this term.
I20250626 14:04:52.572500 3071 leader_election.cc:304] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19; no voters: 24ad911d71b6433f8c7939a1d1f7b782, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:04:52.572788 3127 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 4 FOLLOWER]: Advancing to term 5
I20250626 14:04:52.574465 3127 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 14:04:53.321261 3154 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:53.321403 3154 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:53.321888 3154 leader_election.cc:290] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
I20250626 14:04:53.322634 2990 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07532b7648e4414789ed2b099bad2b19" candidate_term: 6 candidate_status { last_received { term: 4 index: 71 } } ignore_live_leader: false dest_uuid: "81db61eed5584e7e8b0cf40e5441f88e" is_pre_election: true
I20250626 14:04:53.322813 2990 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07532b7648e4414789ed2b099bad2b19 in term 5.
I20250626 14:04:53.323123 3071 leader_election.cc:304] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e; no voters:
W20250626 14:04:53.323283 3071 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
I20250626 14:04:53.323299 3154 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 14:04:53.323364 3154 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:53.323421 3154 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 5 FOLLOWER]: Advancing to term 6
I20250626 14:04:53.324328 3154 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:53.324702 3154 leader_election.cc:290] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 6 election: Requested vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
I20250626 14:04:53.325218 2990 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07532b7648e4414789ed2b099bad2b19" candidate_term: 6 candidate_status { last_received { term: 4 index: 71 } } ignore_live_leader: false dest_uuid: "81db61eed5584e7e8b0cf40e5441f88e"
I20250626 14:04:53.325349 2990 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 5 FOLLOWER]: Advancing to term 6
W20250626 14:04:53.325485 3071 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
I20250626 14:04:53.327163 2990 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07532b7648e4414789ed2b099bad2b19 in term 6.
I20250626 14:04:53.327461 3071 leader_election.cc:304] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e; no voters: 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:53.327658 3154 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 14:04:53.327823 3154 raft_consensus.cc:695] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 6 LEADER]: Becoming Leader. State: Replica: 07532b7648e4414789ed2b099bad2b19, State: Running, Role: LEADER
I20250626 14:04:53.328117 3154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 4.71, Last appended by leader: 71, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:53.329912 3157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07532b7648e4414789ed2b099bad2b19. Latest consensus state: current_term: 6 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:53.330024 3157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:53.330308 3159 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 14:04:53.331977 3159 catalog_manager.cc:671] Loaded metadata for table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]
I20250626 14:04:53.332274 3159 catalog_manager.cc:671] Loaded metadata for table default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843]
I20250626 14:04:53.332513 3159 catalog_manager.cc:671] Loaded metadata for table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]
I20250626 14:04:53.332885 3159 catalog_manager.cc:671] Loaded metadata for table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]
I20250626 14:04:53.333122 3159 catalog_manager.cc:671] Loaded metadata for table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9]
I20250626 14:04:53.333339 3159 catalog_manager.cc:671] Loaded metadata for table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]
I20250626 14:04:53.333567 3159 catalog_manager.cc:671] Loaded metadata for table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]
I20250626 14:04:53.333799 3159 catalog_manager.cc:671] Loaded metadata for table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]
I20250626 14:04:53.334012 3159 catalog_manager.cc:671] Loaded metadata for table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]
I20250626 14:04:53.334214 3159 catalog_manager.cc:671] Loaded metadata for table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]
I20250626 14:04:53.334507 3159 catalog_manager.cc:671] Loaded metadata for table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]
I20250626 14:04:53.335615 3159 tablet_loader.cc:96] loaded metadata for tablet 025415f7a05746e19d5ed199246fc949 (table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a])
I20250626 14:04:53.335948 3159 tablet_loader.cc:96] loaded metadata for tablet 0db0b418e8b8412da3dc098ef6aab940 (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140])
I20250626 14:04:53.336118 3159 tablet_loader.cc:96] loaded metadata for tablet 17a5b09fa0b54d45b835aa64dac8a1af (table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447])
I20250626 14:04:53.336269 3159 tablet_loader.cc:96] loaded metadata for tablet 1e307e729d7d42a68807263959c223a6 (table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb])
I20250626 14:04:53.336411 3159 tablet_loader.cc:96] loaded metadata for tablet 33d002d179974a95aaad3d84814d27f0 (table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75])
I20250626 14:04:53.336866 3159 tablet_loader.cc:96] loaded metadata for tablet 5f5ce7fc87164ae1ace91557275285f8 (table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233])
I20250626 14:04:53.337474 3159 tablet_loader.cc:96] loaded metadata for tablet 730ed7238006463eb08803a70330a43f (table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9])
I20250626 14:04:53.337684 3159 tablet_loader.cc:96] loaded metadata for tablet 7c18e8a737f548c6ab77f739ca17135b (table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80])
I20250626 14:04:53.338091 3159 tablet_loader.cc:96] loaded metadata for tablet b25221169c9c48718276736fe657f378 (table default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843])
I20250626 14:04:53.338598 3159 tablet_loader.cc:96] loaded metadata for tablet dd5f0eb6de154b3c9ffdd6d8f7042a49 (table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03])
I20250626 14:04:53.338764 3159 tablet_loader.cc:96] loaded metadata for tablet f1c96426790242fdb8224137d55939f2 (table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c])
I20250626 14:04:53.339144 3159 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 14:04:53.339818 3159 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:53.339869 3159 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 14:04:53.340735 3159 catalog_manager.cc:1506] Loading token signing keys...
I20250626 14:04:53.341404 3159 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Loaded TSK: 0
I20250626 14:04:53.342401 3159 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 14:04:53.343030 3159 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250626 14:04:53.343601 3159 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 13
I20250626 14:04:53.454501 3081 master_service.cc:948] ReplaceTablet: received request to replace tablet 17a5b09fa0b54d45b835aa64dac8a1af from {username='slave'} at 127.0.0.1:41448
I20250626 14:04:53.454612 3087 master_service.cc:948] ReplaceTablet: received request to replace tablet dd5f0eb6de154b3c9ffdd6d8f7042a49 from {username='slave'} at 127.0.0.1:41448
I20250626 14:04:53.454866 3085 master_service.cc:948] ReplaceTablet: received request to replace tablet 5f5ce7fc87164ae1ace91557275285f8 from {username='slave'} at 127.0.0.1:41448
I20250626 14:04:53.454993 3086 master_service.cc:948] ReplaceTablet: received request to replace tablet 7c18e8a737f548c6ab77f739ca17135b from {username='slave'} at 127.0.0.1:41448
I20250626 14:04:53.455209 3079 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41448:
table { table_name: "default.Table_22f5000aea364833a97948c4478f7e27" } new_table_name: "default.Table_8268b369fb20436fb4e562f6a97eee1d" modify_external_catalogs: true
I20250626 14:04:53.455303 3082 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41448:
table { table_name: "default.Table_42d07c2b16314b1999eea5f3878944fe" } modify_external_catalogs: true
I20250626 14:04:53.455425 3082 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41448:
table { table_name: "default.Table_42d07c2b16314b1999eea5f3878944fe" } modify_external_catalogs: true
I20250626 14:04:53.455606 3083 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41448:
table { table_name: "default.Table_afd26acd01c7422fbb06520af5a4f76c" } new_table_name: "default.Table_4d81e86bd6614c3db77efeaca0daa56f" modify_external_catalogs: true
I20250626 14:04:53.454542 3084 master_service.cc:948] ReplaceTablet: received request to replace tablet 5f5ce7fc87164ae1ace91557275285f8 from {username='slave'} at 127.0.0.1:41448
I20250626 14:04:53.455746 3080 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41448:
name: "default.Table_34cb28e2c8d740aba5c52377533fdfa4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:53.456130 3078 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41448:
name: "default.Table_566f55c5d8c34043a37041fef8cb9f45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:53.459334 2990 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 6 FOLLOWER]: Refusing update from remote peer 07532b7648e4414789ed2b099bad2b19: Log matching property violated. Preceding OpId in replica: term: 3 index: 63. Preceding OpId from leader: term: 6 index: 73. (index mismatch)
W20250626 14:04:53.461144 3071 consensus_peers.cc:487] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 -> Peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Couldn't send request to peer 24ad911d71b6433f8c7939a1d1f7b782. Status: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250626 14:04:53.461594 3140 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 16, previous 16; dumping first 4 out of 8 received events: NotificationEvent(eventId=14, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_72f9307d2f5247a0ae4499c953ea1650, message={"server":"","servicePrincipal":"","db":"default","table":"table_72f9307d2f5247a0ae4499c953ea1650","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_72f9307d2f5247a0ae4499c953ea1650\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_72f9307d2f5247a0ae4499c953ea1650\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_72f9307d2f5247a0ae4499c953ea1650\",\"kudu.table_id\":\"61be84e63b2e478b9da5619fdbdffee9\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=15, eventTime=1750946674, eventType=CREATE_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=ALTER_TABLE, dbName=default, tableName=table_7d5a9b0c3ae440b89dac3ca5dd8e81ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_fb8ebc7aa8894f28b4d79252382f356a","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_fb8ebc7aa8894f28b4d79252382f356a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_fb8ebc7aa8894f28b4d79252382f356a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fb8ebc7aa8894f28b4d79252382f356a\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250626 14:04:53.461771 3163 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 63, Time since last communication: 0.000s
@ 0x7fb146ecedcd google::LogMessage::Fail() at ??:0
@ 0x7fb146ed2b93 google::LogMessage::SendToLog() at ??:0
I20250626 14:04:53.468499 3164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07532b7648e4414789ed2b099bad2b19. Latest consensus state: current_term: 6 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:53.468667 3164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fb146ece7cc google::LogMessage::Flush() at ??:0
I20250626 14:04:53.470175 3161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:53.470324 3161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:53.470418 3164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:53.470521 3164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:53.470723 3168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:53.470798 3168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fb146ecea11 google::LogMessage::~LogMessage() at ??:0
I20250626 14:04:53.472298 3165 mvcc.cc:204] Tried to move back new op lower bound from 7171877656409460736 to 7171877585481977856. Current Snapshot: MvccSnapshot[applied={T|T < 7171877585846472704 or (T in {7171877656402345984})}]
I20250626 14:04:53.473892 3087 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Sending DeleteTablet for 3 replicas of tablet dd5f0eb6de154b3c9ffdd6d8f7042a49
I20250626 14:04:53.473901 3085 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Sending DeleteTablet for 3 replicas of tablet 5f5ce7fc87164ae1ace91557275285f8
I20250626 14:04:53.474411 3084 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41448:
name: "default.Table_fdd6a46b24984290ac6f503f5684b003"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:53.474655 3086 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Sending DeleteTablet for 3 replicas of tablet 7c18e8a737f548c6ab77f739ca17135b
I20250626 14:04:53.475174 3087 catalog_manager.cc:6652] ReplaceTablet: tablet dd5f0eb6de154b3c9ffdd6d8f7042a49 deleted and replaced by tablet aa32cb56ec7346deac266cfb5e8279c5
I20250626 14:04:53.475294 3085 catalog_manager.cc:6652] ReplaceTablet: tablet 5f5ce7fc87164ae1ace91557275285f8 deleted and replaced by tablet 36de2c343f1345e28cb78616a9c25526
I20250626 14:04:53.475482 3085 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41448:
name: "default.Table_213d95d0e97546c298d94c45091826cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:53.475996 3086 catalog_manager.cc:6652] ReplaceTablet: tablet 7c18e8a737f548c6ab77f739ca17135b deleted and replaced by tablet a679d473653e403584d3271e7ce68515
I20250626 14:04:53.476213 3086 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41448:
name: "default.Table_8c238e2636e9491f938a48294f961982"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:53.476704 3081 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Sending DeleteTablet for 3 replicas of tablet 17a5b09fa0b54d45b835aa64dac8a1af
I20250626 14:04:53.475494 3087 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41448:
name: "default.Table_2c8dee0163784a60a8468b53619247f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:53.477008 3081 catalog_manager.cc:6652] ReplaceTablet: tablet 17a5b09fa0b54d45b835aa64dac8a1af deleted and replaced by tablet 94c7591e52cd4d88946f7e98f15e2b94
I20250626 14:04:53.477241 3081 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41448:
name: "default.Table_33ee3521f64d4cefb8856259510e5f0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fb14be67c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fb14be66ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fb14be6630b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250626 14:04:53.485950 1656 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f5ce7fc87164ae1ace91557275285f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 36de2c343f1345e28cb78616a9c25526) from {username='slave'} at 127.0.0.1:35318
I20250626 14:04:53.486131 1658 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c18e8a737f548c6ab77f739ca17135b with delete_type TABLET_DATA_DELETED (replaced by tablet a679d473653e403584d3271e7ce68515) from {username='slave'} at 127.0.0.1:35318
I20250626 14:04:53.486315 1655 tablet_service.cc:1515] Processing DeleteTablet for tablet 17a5b09fa0b54d45b835aa64dac8a1af with delete_type TABLET_DATA_DELETED (replaced by tablet 94c7591e52cd4d88946f7e98f15e2b94) from {username='slave'} at 127.0.0.1:35318
I20250626 14:04:53.486001 1657 tablet_service.cc:1515] Processing DeleteTablet for tablet dd5f0eb6de154b3c9ffdd6d8f7042a49 with delete_type TABLET_DATA_DELETED (replaced by tablet aa32cb56ec7346deac266cfb5e8279c5) from {username='slave'} at 127.0.0.1:35318
I20250626 14:04:53.487823 1657 tablet_service.cc:1468] Processing CreateTablet for tablet f1c96426790242fdb8224137d55939f2 (DEFAULT_TABLE table=default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.488133 1657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c96426790242fdb8224137d55939f2. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fb14be6ba12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56224e693752 std::function<>::operator()() at ??:0
I20250626 14:04:53.491045 1793 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c18e8a737f548c6ab77f739ca17135b with delete_type TABLET_DATA_DELETED (replaced by tablet a679d473653e403584d3271e7ce68515) from {username='slave'} at 127.0.0.1:36806
I20250626 14:04:53.491178 1792 tablet_service.cc:1515] Processing DeleteTablet for tablet dd5f0eb6de154b3c9ffdd6d8f7042a49 with delete_type TABLET_DATA_DELETED (replaced by tablet aa32cb56ec7346deac266cfb5e8279c5) from {username='slave'} at 127.0.0.1:36806
I20250626 14:04:53.491242 1790 tablet_service.cc:1515] Processing DeleteTablet for tablet 17a5b09fa0b54d45b835aa64dac8a1af with delete_type TABLET_DATA_DELETED (replaced by tablet 94c7591e52cd4d88946f7e98f15e2b94) from {username='slave'} at 127.0.0.1:36806
I20250626 14:04:53.491185 1791 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f5ce7fc87164ae1ace91557275285f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 36de2c343f1345e28cb78616a9c25526) from {username='slave'} at 127.0.0.1:36806
I20250626 14:04:53.492254 1791 tablet_service.cc:1468] Processing CreateTablet for tablet f1c96426790242fdb8224137d55939f2 (DEFAULT_TABLE table=default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.492504 1792 tablet_service.cc:1468] Processing CreateTablet for tablet aa32cb56ec7346deac266cfb5e8279c5 (DEFAULT_TABLE table=default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.492534 1791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c96426790242fdb8224137d55939f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.492623 1787 tablet_service.cc:1468] Processing CreateTablet for tablet 025415f7a05746e19d5ed199246fc949 (DEFAULT_TABLE table=default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.492785 1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa32cb56ec7346deac266cfb5e8279c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.492841 1793 tablet_service.cc:1468] Processing CreateTablet for tablet 33d002d179974a95aaad3d84814d27f0 (DEFAULT_TABLE table=default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.492980 1788 tablet_service.cc:1468] Processing CreateTablet for tablet 1e307e729d7d42a68807263959c223a6 (DEFAULT_TABLE table=default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.493054 1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33d002d179974a95aaad3d84814d27f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.493167 1788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e307e729d7d42a68807263959c223a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.495157 1789 tablet_service.cc:1468] Processing CreateTablet for tablet a679d473653e403584d3271e7ce68515 (DEFAULT_TABLE table=default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.495460 1789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a679d473653e403584d3271e7ce68515. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fb1483200a1 kudu::Thread::SuperviseThread() at ??:0
I20250626 14:04:53.495597 1658 tablet_service.cc:1468] Processing CreateTablet for tablet a679d473653e403584d3271e7ce68515 (DEFAULT_TABLE table=default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.495760 1653 tablet_service.cc:1468] Processing CreateTablet for tablet 025415f7a05746e19d5ed199246fc949 (DEFAULT_TABLE table=default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.495846 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a679d473653e403584d3271e7ce68515. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.495970 1653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 025415f7a05746e19d5ed199246fc949. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fb148b0f6db start_thread at ??:0
I20250626 14:04:53.498051 1655 tablet_service.cc:1468] Processing CreateTablet for tablet 33d002d179974a95aaad3d84814d27f0 (DEFAULT_TABLE table=default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.498276 1655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33d002d179974a95aaad3d84814d27f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.498375 1790 tablet_service.cc:1468] Processing CreateTablet for tablet 36de2c343f1345e28cb78616a9c25526 (DEFAULT_TABLE table=default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.498633 1790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36de2c343f1345e28cb78616a9c25526. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.495608 1654 tablet_service.cc:1468] Processing CreateTablet for tablet 1e307e729d7d42a68807263959c223a6 (DEFAULT_TABLE table=default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7fb14608571f clone at ??:0
I20250626 14:04:53.500101 1654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e307e729d7d42a68807263959c223a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.500768 1656 tablet_service.cc:1468] Processing CreateTablet for tablet aa32cb56ec7346deac266cfb5e8279c5 (DEFAULT_TABLE table=default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.501016 1787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 025415f7a05746e19d5ed199246fc949. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.501049 1656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa32cb56ec7346deac266cfb5e8279c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.501070 3188 tablet_bootstrap.cc:492] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:53.502112 3188 tablet_bootstrap.cc:654] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.502849 1657 tablet_service.cc:1468] Processing CreateTablet for tablet 36de2c343f1345e28cb78616a9c25526 (DEFAULT_TABLE table=default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.503094 1657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36de2c343f1345e28cb78616a9c25526. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.503468 3188 tablet_bootstrap.cc:492] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:53.503541 3188 ts_tablet_manager.cc:1397] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:53.503667 3185 tablet_bootstrap.cc:492] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:53.503991 3188 raft_consensus.cc:357] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.504098 3188 raft_consensus.cc:383] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.504132 3188 raft_consensus.cc:738] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.504298 3188 consensus_queue.cc:260] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.504424 1927 tablet_service.cc:1515] Processing DeleteTablet for tablet dd5f0eb6de154b3c9ffdd6d8f7042a49 with delete_type TABLET_DATA_DELETED (replaced by tablet aa32cb56ec7346deac266cfb5e8279c5) from {username='slave'} at 127.0.0.1:34866
I20250626 14:04:53.504724 3188 ts_tablet_manager.cc:1428] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:53.504889 1927 tablet_service.cc:1468] Processing CreateTablet for tablet 36de2c343f1345e28cb78616a9c25526 (DEFAULT_TABLE table=default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.505020 1923 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f5ce7fc87164ae1ace91557275285f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 36de2c343f1345e28cb78616a9c25526) from {username='slave'} at 127.0.0.1:34866
I20250626 14:04:53.505133 1927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36de2c343f1345e28cb78616a9c25526. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.505235 1925 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c18e8a737f548c6ab77f739ca17135b with delete_type TABLET_DATA_DELETED (replaced by tablet a679d473653e403584d3271e7ce68515) from {username='slave'} at 127.0.0.1:34866
I20250626 14:04:53.505355 1926 tablet_service.cc:1515] Processing DeleteTablet for tablet 17a5b09fa0b54d45b835aa64dac8a1af with delete_type TABLET_DATA_DELETED (replaced by tablet 94c7591e52cd4d88946f7e98f15e2b94) from {username='slave'} at 127.0.0.1:34866
I20250626 14:04:53.505573 1928 tablet_service.cc:1468] Processing CreateTablet for tablet f1c96426790242fdb8224137d55939f2 (DEFAULT_TABLE table=default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.505759 1928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c96426790242fdb8224137d55939f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.506026 3188 tablet_bootstrap.cc:492] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:53.507061 3188 tablet_bootstrap.cc:654] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.507838 1924 tablet_service.cc:1468] Processing CreateTablet for tablet 33d002d179974a95aaad3d84814d27f0 (DEFAULT_TABLE table=default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.508064 1924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33d002d179974a95aaad3d84814d27f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.509670 3191 tablet_bootstrap.cc:492] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:53.510015 1918 tablet_service.cc:1468] Processing CreateTablet for tablet 025415f7a05746e19d5ed199246fc949 (DEFAULT_TABLE table=default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.510221 1918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 025415f7a05746e19d5ed199246fc949. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.510334 1920 tablet_service.cc:1468] Processing CreateTablet for tablet aa32cb56ec7346deac266cfb5e8279c5 (DEFAULT_TABLE table=default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.510555 1920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa32cb56ec7346deac266cfb5e8279c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.510718 3191 tablet_bootstrap.cc:654] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.511597 3190 tablet_replica.cc:331] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:53.511715 3190 raft_consensus.cc:2241] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:53.511941 3190 raft_consensus.cc:2270] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:53.512382 3190 ts_tablet_manager.cc:1905] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:53.512559 1922 tablet_service.cc:1468] Processing CreateTablet for tablet a679d473653e403584d3271e7ce68515 (DEFAULT_TABLE table=default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.512741 3191 tablet_bootstrap.cc:492] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:53.512773 1922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a679d473653e403584d3271e7ce68515. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.512812 3191 ts_tablet_manager.cc:1397] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:53.513293 3191 raft_consensus.cc:357] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.513398 3191 raft_consensus.cc:383] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.513432 3191 raft_consensus.cc:738] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.513594 3191 consensus_queue.cc:260] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.514573 3190 ts_tablet_manager.cc:1918] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:53.514639 3190 log.cc:1199] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/dd5f0eb6de154b3c9ffdd6d8f7042a49
I20250626 14:04:53.514899 3190 ts_tablet_manager.cc:1939] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:53.514866 1921 tablet_service.cc:1468] Processing CreateTablet for tablet 1e307e729d7d42a68807263959c223a6 (DEFAULT_TABLE table=default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:53.515082 1921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e307e729d7d42a68807263959c223a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:53.515614 3190 tablet_replica.cc:331] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:53.515744 3190 raft_consensus.cc:2241] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:53.515913 3190 raft_consensus.cc:2270] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:53.516234 3191 ts_tablet_manager.cc:1428] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250626 14:04:53.516270 3190 ts_tablet_manager.cc:1905] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:53.518168 3184 tablet_replica.cc:331] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:53.518311 3184 raft_consensus.cc:2241] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:53.518366 3190 ts_tablet_manager.cc:1918] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 14:04:53.518445 3191 tablet_bootstrap.cc:492] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:53.518515 3184 raft_consensus.cc:2270] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:53.518906 3184 ts_tablet_manager.cc:1905] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:53.519199 3185 tablet_bootstrap.cc:654] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.519454 3191 tablet_bootstrap.cc:654] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.518443 3190 log.cc:1199] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/5f5ce7fc87164ae1ace91557275285f8
I20250626 14:04:53.520251 3186 tablet_replica.cc:331] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:53.520291 3190 ts_tablet_manager.cc:1939] T 5f5ce7fc87164ae1ace91557275285f8 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:53.520407 3186 raft_consensus.cc:2241] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:53.520679 3186 raft_consensus.cc:2270] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:53.520892 3190 tablet_replica.cc:331] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:53.520998 3190 raft_consensus.cc:2241] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:53.521086 3190 raft_consensus.cc:2270] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:53.521207 3186 ts_tablet_manager.cc:1905] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:53.521404 3190 ts_tablet_manager.cc:1905] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:53.523509 3186 ts_tablet_manager.cc:1918] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:53.523577 3186 log.cc:1199] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/7c18e8a737f548c6ab77f739ca17135b
I20250626 14:04:53.523685 3190 ts_tablet_manager.cc:1918] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:53.523754 3190 log.cc:1199] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/7c18e8a737f548c6ab77f739ca17135b
I20250626 14:04:53.523841 3186 ts_tablet_manager.cc:1939] T 7c18e8a737f548c6ab77f739ca17135b P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:53.524008 3190 ts_tablet_manager.cc:1939] T 7c18e8a737f548c6ab77f739ca17135b P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:53.524523 3186 tablet_replica.cc:331] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:53.524610 3190 tablet_replica.cc:331] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:53.524667 3186 raft_consensus.cc:2241] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:53.524709 3190 raft_consensus.cc:2241] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:53.524776 3186 raft_consensus.cc:2270] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:53.524798 3190 raft_consensus.cc:2270] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:53.525132 3190 ts_tablet_manager.cc:1905] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:53.525134 3186 ts_tablet_manager.cc:1905] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:53.525514 3184 ts_tablet_manager.cc:1918] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 14:04:53.525585 3184 log.cc:1199] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/5f5ce7fc87164ae1ace91557275285f8
I20250626 14:04:53.525846 3184 ts_tablet_manager.cc:1939] T 5f5ce7fc87164ae1ace91557275285f8 P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:53.526506 3184 tablet_replica.cc:331] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:53.526618 3184 raft_consensus.cc:2241] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:53.526713 3184 raft_consensus.cc:2270] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:53.527084 3184 ts_tablet_manager.cc:1905] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:53.527145 3190 ts_tablet_manager.cc:1918] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:53.527199 3190 log.cc:1199] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/17a5b09fa0b54d45b835aa64dac8a1af
I20250626 14:04:53.527287 3186 ts_tablet_manager.cc:1918] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:53.527364 3186 log.cc:1199] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/dd5f0eb6de154b3c9ffdd6d8f7042a49
I20250626 14:04:53.527446 3190 ts_tablet_manager.cc:1939] T 17a5b09fa0b54d45b835aa64dac8a1af P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:53.527647 3186 ts_tablet_manager.cc:1939] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:53.528188 3186 tablet_replica.cc:331] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:53.528294 3186 raft_consensus.cc:2241] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:53.528412 3186 raft_consensus.cc:2270] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:53.528834 3186 ts_tablet_manager.cc:1905] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:53.529573 3185 tablet_bootstrap.cc:492] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:53.529649 3185 ts_tablet_manager.cc:1397] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20250626 14:04:53.530122 3185 raft_consensus.cc:357] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:53.530221 3185 raft_consensus.cc:383] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.530277 3185 raft_consensus.cc:738] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.530433 3184 ts_tablet_manager.cc:1918] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:53.530498 3184 log.cc:1199] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/7c18e8a737f548c6ab77f739ca17135b
I20250626 14:04:53.530455 3185 consensus_queue.cc:260] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:53.530737 3186 ts_tablet_manager.cc:1918] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:53.530766 3185 ts_tablet_manager.cc:1428] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:53.530833 3186 log.cc:1199] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/17a5b09fa0b54d45b835aa64dac8a1af
I20250626 14:04:53.530941 3185 tablet_bootstrap.cc:492] T 025415f7a05746e19d5ed199246fc949 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:53.531064 3186 ts_tablet_manager.cc:1939] T 17a5b09fa0b54d45b835aa64dac8a1af P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:53.531221 3184 ts_tablet_manager.cc:1939] T 7c18e8a737f548c6ab77f739ca17135b P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:53.531653 3186 tablet_replica.cc:331] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:53.531769 3186 raft_consensus.cc:2241] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:53.531908 3184 tablet_replica.cc:331] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:53.531945 3186 raft_consensus.cc:2270] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:53.532013 3184 raft_consensus.cc:2241] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:53.532222 3184 raft_consensus.cc:2270] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:53.532362 3186 ts_tablet_manager.cc:1905] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:53.532604 3184 ts_tablet_manager.cc:1905] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:53.533151 3188 tablet_bootstrap.cc:492] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:53.533231 3188 ts_tablet_manager.cc:1397] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20250626 14:04:53.533319 3185 tablet_bootstrap.cc:654] T 025415f7a05746e19d5ed199246fc949 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.533712 3188 raft_consensus.cc:357] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:53.533818 3188 raft_consensus.cc:383] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.533849 3188 raft_consensus.cc:738] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.533974 3188 consensus_queue.cc:260] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:53.534269 3188 ts_tablet_manager.cc:1428] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 14:04:53.534466 3188 tablet_bootstrap.cc:492] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:53.535110 3191 tablet_bootstrap.cc:492] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:53.535197 3191 ts_tablet_manager.cc:1397] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20250626 14:04:53.535692 3188 tablet_bootstrap.cc:654] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.535779 3191 raft_consensus.cc:357] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:53.535887 3191 raft_consensus.cc:383] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.535928 3191 raft_consensus.cc:738] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.536062 3191 consensus_queue.cc:260] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:53.536345 3191 ts_tablet_manager.cc:1428] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:53.536527 3191 tablet_bootstrap.cc:492] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:53.537535 3191 tablet_bootstrap.cc:654] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.538384 3186 ts_tablet_manager.cc:1918] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250626 14:04:53.538452 3186 log.cc:1199] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/5f5ce7fc87164ae1ace91557275285f8
I20250626 14:04:53.538738 3186 ts_tablet_manager.cc:1939] T 5f5ce7fc87164ae1ace91557275285f8 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:53.538789 3184 ts_tablet_manager.cc:1918] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:53.538861 3184 log.cc:1199] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/17a5b09fa0b54d45b835aa64dac8a1af
I20250626 14:04:53.539104 3184 ts_tablet_manager.cc:1939] T 17a5b09fa0b54d45b835aa64dac8a1af P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:53.540109 3188 tablet_bootstrap.cc:492] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:53.540194 3188 ts_tablet_manager.cc:1397] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250626 14:04:53.540578 3184 tablet_replica.cc:331] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:53.540737 3184 raft_consensus.cc:2241] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:53.540722 3188 raft_consensus.cc:357] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.540822 3188 raft_consensus.cc:383] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.540848 3184 raft_consensus.cc:2270] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:53.540854 3188 raft_consensus.cc:738] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.540983 3188 consensus_queue.cc:260] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.541218 3184 ts_tablet_manager.cc:1905] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:53.541267 3188 ts_tablet_manager.cc:1428] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:53.541494 3188 tablet_bootstrap.cc:492] T 33d002d179974a95aaad3d84814d27f0 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:53.542484 3188 tablet_bootstrap.cc:654] T 33d002d179974a95aaad3d84814d27f0 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.543256 3184 ts_tablet_manager.cc:1918] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:53.543717 3191 tablet_bootstrap.cc:492] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:53.543819 3191 ts_tablet_manager.cc:1397] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20250626 14:04:53.543949 3185 tablet_bootstrap.cc:492] T 025415f7a05746e19d5ed199246fc949 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:53.544034 3185 ts_tablet_manager.cc:1397] T 025415f7a05746e19d5ed199246fc949 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250626 14:04:53.544385 3191 raft_consensus.cc:357] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.544492 3191 raft_consensus.cc:383] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.544549 3191 raft_consensus.cc:738] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.544674 3191 consensus_queue.cc:260] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.544643 3185 raft_consensus.cc:357] T 025415f7a05746e19d5ed199246fc949 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.544759 3185 raft_consensus.cc:383] T 025415f7a05746e19d5ed199246fc949 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.544795 3185 raft_consensus.cc:738] T 025415f7a05746e19d5ed199246fc949 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.544926 3191 ts_tablet_manager.cc:1428] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:53.544930 3185 consensus_queue.cc:260] T 025415f7a05746e19d5ed199246fc949 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.545130 3191 tablet_bootstrap.cc:492] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:53.545225 3185 ts_tablet_manager.cc:1428] T 025415f7a05746e19d5ed199246fc949 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:53.545436 3185 tablet_bootstrap.cc:492] T 33d002d179974a95aaad3d84814d27f0 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:53.546384 3191 tablet_bootstrap.cc:654] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.546690 3185 tablet_bootstrap.cc:654] T 33d002d179974a95aaad3d84814d27f0 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.547273 3184 log.cc:1199] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/dd5f0eb6de154b3c9ffdd6d8f7042a49
I20250626 14:04:53.547626 3184 ts_tablet_manager.cc:1939] T dd5f0eb6de154b3c9ffdd6d8f7042a49 P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:53.549032 3188 tablet_bootstrap.cc:492] T 33d002d179974a95aaad3d84814d27f0 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:53.549127 3188 ts_tablet_manager.cc:1397] T 33d002d179974a95aaad3d84814d27f0 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250626 14:04:53.549513 2906 raft_consensus.cc:491] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:53.549597 2906 raft_consensus.cc:513] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:53.549656 3188 raft_consensus.cc:357] T 33d002d179974a95aaad3d84814d27f0 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.549754 3188 raft_consensus.cc:383] T 33d002d179974a95aaad3d84814d27f0 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.549782 3188 raft_consensus.cc:738] T 33d002d179974a95aaad3d84814d27f0 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.549898 3188 consensus_queue.cc:260] T 33d002d179974a95aaad3d84814d27f0 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.550084 2906 leader_election.cc:290] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:53.550191 3188 ts_tablet_manager.cc:1428] T 33d002d179974a95aaad3d84814d27f0 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:53.550380 3188 tablet_bootstrap.cc:492] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:53.551384 3188 tablet_bootstrap.cc:654] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.551398 3191 tablet_bootstrap.cc:492] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:53.551478 3191 ts_tablet_manager.cc:1397] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20250626 14:04:53.551501 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1c96426790242fdb8224137d55939f2" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:53.551615 1813 raft_consensus.cc:2466] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:53.551920 3191 raft_consensus.cc:357] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.552016 3191 raft_consensus.cc:383] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.552045 3191 raft_consensus.cc:738] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.552181 3191 consensus_queue.cc:260] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.552441 3191 ts_tablet_manager.cc:1428] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 14:04:53.552654 3191 tablet_bootstrap.cc:492] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:53.552453 1883 leader_election.cc:304] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:53.552932 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1c96426790242fdb8224137d55939f2" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:53.553056 1677 raft_consensus.cc:2466] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:53.553625 3185 tablet_bootstrap.cc:492] T 33d002d179974a95aaad3d84814d27f0 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:53.553648 3191 tablet_bootstrap.cc:654] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.553704 3185 ts_tablet_manager.cc:1397] T 33d002d179974a95aaad3d84814d27f0 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250626 14:04:53.554198 3185 raft_consensus.cc:357] T 33d002d179974a95aaad3d84814d27f0 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.554323 3185 raft_consensus.cc:383] T 33d002d179974a95aaad3d84814d27f0 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.554361 3185 raft_consensus.cc:738] T 33d002d179974a95aaad3d84814d27f0 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.554474 3185 consensus_queue.cc:260] T 33d002d179974a95aaad3d84814d27f0 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.554762 3185 ts_tablet_manager.cc:1428] T 33d002d179974a95aaad3d84814d27f0 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:53.554919 3185 tablet_bootstrap.cc:492] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:53.555042 3188 tablet_bootstrap.cc:492] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:53.555117 3188 ts_tablet_manager.cc:1397] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250626 14:04:53.555943 3185 tablet_bootstrap.cc:654] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.570351 3188 raft_consensus.cc:357] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.570467 3188 raft_consensus.cc:383] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.570489 3188 raft_consensus.cc:738] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.570617 3188 consensus_queue.cc:260] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.571883 3188 ts_tablet_manager.cc:1428] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 14:04:53.572166 3188 tablet_bootstrap.cc:492] T 025415f7a05746e19d5ed199246fc949 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:53.573650 3188 tablet_bootstrap.cc:654] T 025415f7a05746e19d5ed199246fc949 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.574527 3185 tablet_bootstrap.cc:492] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:53.574656 3185 ts_tablet_manager.cc:1397] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20250626 14:04:53.574815 3188 tablet_bootstrap.cc:492] T 025415f7a05746e19d5ed199246fc949 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:53.574880 3188 ts_tablet_manager.cc:1397] T 025415f7a05746e19d5ed199246fc949 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:53.575233 3185 raft_consensus.cc:357] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.555279 2906 raft_consensus.cc:2802] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:53.582993 2906 raft_consensus.cc:491] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:53.583101 2906 raft_consensus.cc:3058] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.575294 3188 raft_consensus.cc:357] T 025415f7a05746e19d5ed199246fc949 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.583505 3188 raft_consensus.cc:383] T 025415f7a05746e19d5ed199246fc949 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.583575 3188 raft_consensus.cc:738] T 025415f7a05746e19d5ed199246fc949 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.583742 3188 consensus_queue.cc:260] T 025415f7a05746e19d5ed199246fc949 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.584095 3188 ts_tablet_manager.cc:1428] T 025415f7a05746e19d5ed199246fc949 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250626 14:04:53.575349 3185 raft_consensus.cc:383] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.584352 3188 tablet_bootstrap.cc:492] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:53.584321 3185 raft_consensus.cc:738] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.584405 2906 raft_consensus.cc:513] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:53.584604 3185 consensus_queue.cc:260] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.584903 2906 leader_election.cc:290] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Requested vote from peers 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:53.584944 3185 ts_tablet_manager.cc:1428] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250626 14:04:53.557380 3191 tablet_bootstrap.cc:492] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:53.585068 3191 ts_tablet_manager.cc:1397] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.002s
I20250626 14:04:53.585170 3185 tablet_bootstrap.cc:492] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:53.585592 3188 tablet_bootstrap.cc:654] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.585592 3191 raft_consensus.cc:357] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.585711 3191 raft_consensus.cc:383] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.585745 3191 raft_consensus.cc:738] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.585846 3191 consensus_queue.cc:260] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.586153 3191 ts_tablet_manager.cc:1428] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:53.586357 3191 tablet_bootstrap.cc:492] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:53.586364 3185 tablet_bootstrap.cc:654] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.587330 3191 tablet_bootstrap.cc:654] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.588415 3185 tablet_bootstrap.cc:492] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:53.588516 3185 ts_tablet_manager.cc:1397] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:53.588728 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1c96426790242fdb8224137d55939f2" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:53.588843 1677 raft_consensus.cc:3058] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.589061 3185 raft_consensus.cc:357] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.589143 3185 raft_consensus.cc:383] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.589170 3185 raft_consensus.cc:738] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.589285 3185 consensus_queue.cc:260] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.589638 3185 ts_tablet_manager.cc:1428] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:53.589876 1677 raft_consensus.cc:2466] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:53.589929 3185 tablet_bootstrap.cc:492] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:53.591145 3185 tablet_bootstrap.cc:654] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.591671 1884 leader_election.cc:304] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:53.592016 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1c96426790242fdb8224137d55939f2" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:53.592164 1813 raft_consensus.cc:3058] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.592762 3218 raft_consensus.cc:2802] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:53.592875 3218 raft_consensus.cc:695] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Running, Role: LEADER
I20250626 14:04:53.593283 1813 raft_consensus.cc:2466] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:53.594669 3191 tablet_bootstrap.cc:492] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:53.594712 3188 tablet_bootstrap.cc:492] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:53.594765 3191 ts_tablet_manager.cc:1397] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20250626 14:04:53.594800 3188 ts_tablet_manager.cc:1397] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250626 14:04:53.595041 3185 tablet_bootstrap.cc:492] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:53.595117 3185 ts_tablet_manager.cc:1397] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20250626 14:04:53.595216 3191 raft_consensus.cc:357] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.595335 3191 raft_consensus.cc:383] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.595289 3188 raft_consensus.cc:357] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.595366 3191 raft_consensus.cc:738] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.595381 3188 raft_consensus.cc:383] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.595414 3188 raft_consensus.cc:738] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.595479 3191 consensus_queue.cc:260] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.595535 3188 consensus_queue.cc:260] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.595574 3185 raft_consensus.cc:357] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.595660 3185 raft_consensus.cc:383] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.595691 3185 raft_consensus.cc:738] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.595762 3191 ts_tablet_manager.cc:1428] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 14:04:53.595834 3188 ts_tablet_manager.cc:1428] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:53.595811 3185 consensus_queue.cc:260] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.595932 3191 tablet_bootstrap.cc:492] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:53.596068 3185 ts_tablet_manager.cc:1428] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:53.596244 3185 tablet_bootstrap.cc:492] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:53.597013 3191 tablet_bootstrap.cc:654] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.597764 3185 tablet_bootstrap.cc:654] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:53.599395 3191 tablet_bootstrap.cc:492] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:53.599428 3218 consensus_queue.cc:237] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:53.599517 3191 ts_tablet_manager.cc:1397] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250626 14:04:53.600014 3191 raft_consensus.cc:357] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.600164 3191 raft_consensus.cc:383] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.600243 3191 raft_consensus.cc:738] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.600281 3185 tablet_bootstrap.cc:492] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:53.600405 3191 consensus_queue.cc:260] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.600479 3185 ts_tablet_manager.cc:1397] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20250626 14:04:53.600744 3191 ts_tablet_manager.cc:1428] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:53.601788 3185 raft_consensus.cc:357] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.601961 3185 raft_consensus.cc:383] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:53.602046 3185 raft_consensus.cc:738] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:53.602206 3185 consensus_queue.cc:260] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.602550 3185 ts_tablet_manager.cc:1428] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 14:04:53.630721 2947 raft_consensus.cc:491] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:53.630844 2947 raft_consensus.cc:513] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.631414 2947 leader_election.cc:290] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:53.631878 1948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa32cb56ec7346deac266cfb5e8279c5" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:53.631879 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa32cb56ec7346deac266cfb5e8279c5" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:53.632018 1813 raft_consensus.cc:2466] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 0.
I20250626 14:04:53.632018 1948 raft_consensus.cc:2466] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 0.
I20250626 14:04:53.632380 1613 leader_election.cc:304] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:53.632683 2947 raft_consensus.cc:2802] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:53.632737 2947 raft_consensus.cc:491] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:53.632783 2947 raft_consensus.cc:3058] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.633628 2947 raft_consensus.cc:513] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.633963 2947 leader_election.cc:290] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 election: Requested vote from peers 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:53.634363 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa32cb56ec7346deac266cfb5e8279c5" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:53.634356 1948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa32cb56ec7346deac266cfb5e8279c5" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:53.634482 1948 raft_consensus.cc:3058] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.634480 1813 raft_consensus.cc:3058] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.635425 1948 raft_consensus.cc:2466] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
I20250626 14:04:53.635414 1813 raft_consensus.cc:2466] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
I20250626 14:04:53.635753 1613 leader_election.cc:304] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:53.635937 2947 raft_consensus.cc:2802] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:53.636049 2947 raft_consensus.cc:695] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [term 1 LEADER]: Becoming Leader. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Running, Role: LEADER
I20250626 14:04:53.636205 2947 consensus_queue.cc:237] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.698287 2948 master_runner.cc:186] Error getting master registration for 127.1.11.254:36707: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111),
I20250626 14:04:53.698426 2948 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:04:53.703346 1883 connection.cc:537] server connection from 127.0.0.1:34866 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:53.703387 1748 connection.cc:537] server connection from 127.0.0.1:36806 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:53.703528 1883 connection.cc:537] client connection to 127.1.11.253:44447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:53.703544 1748 connection.cc:537] client connection to 127.1.11.253:44447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:53.703747 1613 connection.cc:537] client connection to 127.1.11.253:44447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:53.703898 1613 connection.cc:537] server connection from 127.0.0.1:35318 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:53.704185 2003 connection.cc:537] client connection to 127.1.11.253:44447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:53.704691 2003 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.11.253:44447)
I20250626 14:04:53.704988 3218 raft_consensus.cc:491] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 14:04:53.705056 2003 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.11.253:44447)
W20250626 14:04:53.705158 2003 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.11.253:44447)
I20250626 14:04:53.705096 3218 raft_consensus.cc:513] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
W20250626 14:04:53.705258 2003 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.11.253:44447)
I20250626 14:04:53.705525 3218 leader_election.cc:290] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:53.705951 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a679d473653e403584d3271e7ce68515" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:53.706087 1813 raft_consensus.cc:2466] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
W20250626 14:04:53.706372 2811 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 14:04:53.705380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.11.253:44447 (local address 127.1.11.194:56339)
0626 14:04:53.705773 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0626 14:04:53.705786 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0626 14:04:53.705867 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0626 14:04:53.706218 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":314}
W20250626 14:04:53.706840 3055 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 14:04:53.704475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.11.253:44447 (local address 127.1.11.193:58093)
0626 14:04:53.704990 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0626 14:04:53.705009 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0626 14:04:53.705106 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0626 14:04:53.706746 (+ 1640us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":411}
W20250626 14:04:53.706892 3153 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 14:04:53.704756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.11.253:44447 (local address 127.1.11.195:38543)
0626 14:04:53.705465 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0626 14:04:53.705479 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0626 14:04:53.705569 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0626 14:04:53.706321 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":625}
I20250626 14:04:53.706933 1883 leader_election.cc:304] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:53.707141 3218 raft_consensus.cc:2802] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:53.707209 3218 raft_consensus.cc:491] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:53.707262 3218 raft_consensus.cc:3058] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.707266 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a679d473653e403584d3271e7ce68515" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:53.707417 1677 raft_consensus.cc:2466] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:53.708249 3218 raft_consensus.cc:513] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.708629 3218 leader_election.cc:290] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Requested vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:53.708945 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a679d473653e403584d3271e7ce68515" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:53.708976 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a679d473653e403584d3271e7ce68515" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:53.709069 1677 raft_consensus.cc:3058] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.709071 1813 raft_consensus.cc:3058] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.709837 1677 raft_consensus.cc:2466] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:53.709853 1813 raft_consensus.cc:2466] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:53.710124 1884 leader_election.cc:304] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:53.710343 3218 raft_consensus.cc:2802] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:53.710448 3218 raft_consensus.cc:695] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Running, Role: LEADER
I20250626 14:04:53.710618 3218 consensus_queue.cc:237] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.711643 3218 raft_consensus.cc:491] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:53.711727 3218 raft_consensus.cc:513] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.712029 3218 leader_election.cc:290] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:53.712301 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36de2c343f1345e28cb78616a9c25526" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:53.712375 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36de2c343f1345e28cb78616a9c25526" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:53.712425 1813 raft_consensus.cc:2466] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:53.712476 1677 raft_consensus.cc:2466] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:53.712767 1884 leader_election.cc:304] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:53.712932 3218 raft_consensus.cc:2802] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:53.712992 3218 raft_consensus.cc:491] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:53.713024 3218 raft_consensus.cc:3058] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.713747 3218 raft_consensus.cc:513] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.714056 3218 leader_election.cc:290] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Requested vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:53.714329 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36de2c343f1345e28cb78616a9c25526" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:53.714355 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36de2c343f1345e28cb78616a9c25526" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:53.714443 1813 raft_consensus.cc:3058] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.714450 1677 raft_consensus.cc:3058] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.715346 1677 raft_consensus.cc:2466] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:53.715415 1813 raft_consensus.cc:2466] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:53.715664 1883 leader_election.cc:304] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:53.715806 3218 raft_consensus.cc:2802] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:53.715893 3218 raft_consensus.cc:695] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Running, Role: LEADER
I20250626 14:04:53.716377 3218 consensus_queue.cc:237] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.732604 3218 raft_consensus.cc:491] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:53.732736 3218 raft_consensus.cc:513] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.733163 3218 leader_election.cc:290] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:53.733587 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33d002d179974a95aaad3d84814d27f0" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:53.733587 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33d002d179974a95aaad3d84814d27f0" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:53.733741 1813 raft_consensus.cc:2466] T 33d002d179974a95aaad3d84814d27f0 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:53.733741 1677 raft_consensus.cc:2466] T 33d002d179974a95aaad3d84814d27f0 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:53.734047 1884 leader_election.cc:304] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:53.734200 3218 raft_consensus.cc:2802] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:53.734297 3218 raft_consensus.cc:491] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:53.734352 3218 raft_consensus.cc:3058] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.735196 3218 raft_consensus.cc:513] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.735731 3218 leader_election.cc:290] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Requested vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:53.735858 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33d002d179974a95aaad3d84814d27f0" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:53.735986 1813 raft_consensus.cc:3058] T 33d002d179974a95aaad3d84814d27f0 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.736114 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33d002d179974a95aaad3d84814d27f0" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:53.736249 1677 raft_consensus.cc:3058] T 33d002d179974a95aaad3d84814d27f0 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.737023 1813 raft_consensus.cc:2466] T 33d002d179974a95aaad3d84814d27f0 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:53.737241 1677 raft_consensus.cc:2466] T 33d002d179974a95aaad3d84814d27f0 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:53.737334 1883 leader_election.cc:304] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:53.737630 3218 raft_consensus.cc:2802] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:53.737715 3218 raft_consensus.cc:695] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Running, Role: LEADER
I20250626 14:04:53.737864 3218 consensus_queue.cc:237] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.832458 3218 raft_consensus.cc:491] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:53.832624 3218 raft_consensus.cc:513] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.833027 3218 leader_election.cc:290] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:53.833364 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e307e729d7d42a68807263959c223a6" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:53.833420 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e307e729d7d42a68807263959c223a6" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:53.833500 1813 raft_consensus.cc:2466] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:53.833560 1677 raft_consensus.cc:2466] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:53.833762 1883 leader_election.cc:304] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:53.833976 3218 raft_consensus.cc:2802] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:53.834044 3218 raft_consensus.cc:491] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:53.834081 3218 raft_consensus.cc:3058] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.834926 3218 raft_consensus.cc:513] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.835238 3218 leader_election.cc:290] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Requested vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:53.835521 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e307e729d7d42a68807263959c223a6" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:53.835574 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e307e729d7d42a68807263959c223a6" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:53.835635 1813 raft_consensus.cc:3058] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.835788 1677 raft_consensus.cc:3058] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.836616 1813 raft_consensus.cc:2466] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:53.836624 1677 raft_consensus.cc:2466] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:53.836912 1883 leader_election.cc:304] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:53.837131 3218 raft_consensus.cc:2802] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:53.837235 3218 raft_consensus.cc:695] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Running, Role: LEADER
I20250626 14:04:53.837381 3218 consensus_queue.cc:237] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.900408 3218 raft_consensus.cc:491] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:53.900573 3218 raft_consensus.cc:513] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.900966 3218 leader_election.cc:290] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:53.901365 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025415f7a05746e19d5ed199246fc949" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:53.901404 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025415f7a05746e19d5ed199246fc949" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:53.901515 1813 raft_consensus.cc:2466] T 025415f7a05746e19d5ed199246fc949 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:53.901578 1677 raft_consensus.cc:2466] T 025415f7a05746e19d5ed199246fc949 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:53.901824 1883 leader_election.cc:304] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:53.901968 3218 raft_consensus.cc:2802] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:53.902024 3218 raft_consensus.cc:491] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:53.902071 3218 raft_consensus.cc:3058] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.902885 3218 raft_consensus.cc:513] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.903254 3218 leader_election.cc:290] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Requested vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:53.903590 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025415f7a05746e19d5ed199246fc949" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:53.903628 1677 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025415f7a05746e19d5ed199246fc949" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:53.903708 1813 raft_consensus.cc:3058] T 025415f7a05746e19d5ed199246fc949 P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.903740 1677 raft_consensus.cc:3058] T 025415f7a05746e19d5ed199246fc949 P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:53.904458 1813 raft_consensus.cc:2466] T 025415f7a05746e19d5ed199246fc949 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:53.904465 1677 raft_consensus.cc:2466] T 025415f7a05746e19d5ed199246fc949 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:53.904778 1884 leader_election.cc:304] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:53.904932 3218 raft_consensus.cc:2802] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:53.905017 3218 raft_consensus.cc:695] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Running, Role: LEADER
I20250626 14:04:53.905185 3218 consensus_queue.cc:237] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:53.999058 3232 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 07532b7648e4414789ed2b099bad2b19)
I20250626 14:04:53.999217 3232 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:53.999697 3232 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
W20250626 14:04:54.000716 2964 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
W20250626 14:04:54.000816 2966 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:54.000905 2966 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:54.001040 3232 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 14:04:54.104015 3218 consensus_queue.cc:1035] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:54.108394 3237 consensus_queue.cc:1035] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:54.111249 3237 consensus_queue.cc:1035] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:54.114195 2946 consensus_queue.cc:1035] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:54.118018 3237 consensus_queue.cc:1035] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:54.121438 2947 consensus_queue.cc:1035] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:54.196218 3237 consensus_queue.cc:1035] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:54.199818 3237 consensus_queue.cc:1035] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:54.216753 3237 consensus_queue.cc:1035] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:54.221340 2906 consensus_queue.cc:1035] T 33d002d179974a95aaad3d84814d27f0 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:54.287545 2906 consensus_queue.cc:1035] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:54.291589 2906 consensus_queue.cc:1035] T 025415f7a05746e19d5ed199246fc949 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:54.321699 2906 consensus_queue.cc:1035] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:54.324786 2906 consensus_queue.cc:1035] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:54.539883 3254 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 07532b7648e4414789ed2b099bad2b19)
I20250626 14:04:54.540032 3254 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:54.540551 3254 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
W20250626 14:04:54.541342 2964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20250626 14:04:54.542284 2964 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
W20250626 14:04:54.543145 2966 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:54.543219 2966 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:54.543373 3254 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 14:04:55.131074 3258 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 07532b7648e4414789ed2b099bad2b19)
I20250626 14:04:55.131206 3258 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:55.131675 3258 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
W20250626 14:04:55.132905 2966 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:04:55.133004 2964 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
I20250626 14:04:55.133069 2964 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:55.133191 3258 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 14:04:55.323056 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.253:44447
--webserver_interface=127.1.11.253
--webserver_port=37937
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250626 14:04:55.434628 3262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:55.434926 3262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:55.434988 3262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:04:55.438817 3262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:04:55.438879 3262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:55.438900 3262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:04:55.438917 3262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:04:55.443173 3262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.253:44447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.11.253
--webserver_port=37937
--never_fsync=true
--heap_profile_path=/tmp/kudu.3262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:55.444324 3262 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:55.445506 3262 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:04:55.452731 3273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:55.452723 3270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:55.452723 3262 server_base.cc:1048] running on GCE node
W20250626 14:04:55.452755 3271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:55.453670 3262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:55.454617 3262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:55.455804 3262 hybrid_clock.cc:648] HybridClock initialized: now 1750946695455791 us; error 27 us; skew 500 ppm
I20250626 14:04:55.463478 3262 webserver.cc:469] Webserver started at http://127.1.11.253:37937/ using document root <none> and password file <none>
I20250626 14:04:55.464103 3262 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:55.464170 3262 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:55.467859 3262 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:55.469900 3279 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:55.470887 3262 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250626 14:04:55.471004 3262 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
uuid: "07532b7648e4414789ed2b099bad2b19"
format_stamp: "Formatted at 2025-06-26 14:04:30 on dist-test-slave-rptj"
I20250626 14:04:55.471515 3262 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:04:55.484335 3262 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:55.485109 3262 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:55.485297 3262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:55.493315 3331 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.253:44447 every 8 connection(s)
I20250626 14:04:55.493315 3262 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.253:44447
I20250626 14:04:55.494593 3262 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
I20250626 14:04:55.498440 3332 sys_catalog.cc:263] Verifying existing consensus state
I20250626 14:04:55.499384 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 3262
I20250626 14:04:55.500119 3332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Bootstrap starting.
I20250626 14:04:55.510243 3332 log.cc:826] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Log is configured to *not* fsync() on all Append() calls
I20250626 14:04:55.547720 3332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Bootstrap replayed 1/1 log segments. Stats: ops{read=78 overwritten=0 applied=77 ignored=0} inserts{seen=53 ignored=0} mutations{seen=69 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250626 14:04:55.548261 3332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Bootstrap complete.
I20250626 14:04:55.551604 3332 raft_consensus.cc:357] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:55.553037 3332 raft_consensus.cc:738] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07532b7648e4414789ed2b099bad2b19, State: Initialized, Role: FOLLOWER
I20250626 14:04:55.553475 3332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 77, Last appended: 6.78, Last appended by leader: 78, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:55.553890 3337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:55.553997 3337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:55.554481 3332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:04:55.560751 3351 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:55.560889 3351 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: loading cluster ID for follower catalog manager: success
I20250626 14:04:55.559418 3262 master_runner.cc:186] Error getting master registration for 127.1.11.254:36707: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111),
I20250626 14:04:55.561269 3262 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:04:55.563722 3351 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: acquiring CA information for follower catalog manager: success
I20250626 14:04:55.565708 3351 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 14:04:55.632602 1861 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:55.635020 3296 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e517893e13f546c0b40d976633b49651" instance_seqno: 1750946671821196) as {username='slave'} at 127.1.11.194:52353; Asking this server to re-register.
I20250626 14:04:55.635612 1861 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:55.636932 3296 ts_manager.cc:194] Registered new tserver with Master: e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:55.904197 1996 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:55.905467 3296 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" instance_seqno: 1750946672002466) as {username='slave'} at 127.1.11.195:44279; Asking this server to re-register.
I20250626 14:04:55.905901 1996 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:55.906780 3296 ts_manager.cc:194] Registered new tserver with Master: 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:55.940474 1726 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:55.941777 3296 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" instance_seqno: 1750946671637378) as {username='slave'} at 127.1.11.193:44297; Asking this server to re-register.
I20250626 14:04:55.942219 1726 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:55.943125 3296 ts_manager.cc:194] Registered new tserver with Master: 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
W20250626 14:04:56.103221 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 21: this message will repeat every 5th retry.
I20250626 14:04:56.106770 3361 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:56.106952 3361 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:56.108307 3361 leader_election.cc:290] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:04:56.108952 3280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
W20250626 14:04:56.109905 3280 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
I20250626 14:04:56.112125 2990 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07532b7648e4414789ed2b099bad2b19" candidate_term: 7 candidate_status { last_received { term: 6 index: 78 } } ignore_live_leader: false dest_uuid: "81db61eed5584e7e8b0cf40e5441f88e" is_pre_election: true
I20250626 14:04:56.112265 2990 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07532b7648e4414789ed2b099bad2b19 in term 6.
I20250626 14:04:56.112706 3280 leader_election.cc:304] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e; no voters: 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:56.112952 3361 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 14:04:56.113034 3361 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:56.113106 3361 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 6 FOLLOWER]: Advancing to term 7
I20250626 14:04:56.114764 3361 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:56.115092 3361 leader_election.cc:290] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 7 election: Requested vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
I20250626 14:04:56.115697 2990 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07532b7648e4414789ed2b099bad2b19" candidate_term: 7 candidate_status { last_received { term: 6 index: 78 } } ignore_live_leader: false dest_uuid: "81db61eed5584e7e8b0cf40e5441f88e"
I20250626 14:04:56.115855 2990 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 6 FOLLOWER]: Advancing to term 7
W20250626 14:04:56.116082 3280 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
I20250626 14:04:56.116804 2990 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07532b7648e4414789ed2b099bad2b19 in term 7.
I20250626 14:04:56.117102 3280 leader_election.cc:304] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e; no voters: 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:56.117313 3361 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 14:04:56.117522 3361 raft_consensus.cc:695] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 7 LEADER]: Becoming Leader. State: Replica: 07532b7648e4414789ed2b099bad2b19, State: Running, Role: LEADER
I20250626 14:04:56.117815 3361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 6.78, Last appended by leader: 78, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:56.119391 3366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07532b7648e4414789ed2b099bad2b19. Latest consensus state: current_term: 7 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:56.119511 3366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: LEADER
W20250626 14:04:56.237639 1748 consensus_peers.cc:487] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3. This is attempt 21: this message will repeat every 5th retry.
I20250626 14:04:56.530629 2990 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Refusing update from remote peer 07532b7648e4414789ed2b099bad2b19: Log matching property violated. Preceding OpId in replica: term: 6 index: 78. Preceding OpId from leader: term: 7 index: 79. (index mismatch)
I20250626 14:04:56.531172 3366 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20250626 14:04:56.532558 3369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07532b7648e4414789ed2b099bad2b19. Latest consensus state: current_term: 7 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:56.532706 3369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:56.534124 3366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:56.534286 3366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:56.534277 3369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:56.534372 3369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: FOLLOWER
W20250626 14:04:56.535615 3280 consensus_peers.cc:487] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 -> Peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Couldn't send request to peer 24ad911d71b6433f8c7939a1d1f7b782. Status: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 14:04:56.635748 3368 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 14:04:56.637738 3368 catalog_manager.cc:671] Loaded metadata for table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]
I20250626 14:04:56.638085 3368 catalog_manager.cc:671] Loaded metadata for table default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843]
I20250626 14:04:56.638348 3368 catalog_manager.cc:671] Loaded metadata for table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]
I20250626 14:04:56.638759 3368 catalog_manager.cc:671] Loaded metadata for table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]
I20250626 14:04:56.639006 3368 catalog_manager.cc:671] Loaded metadata for table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9]
I20250626 14:04:56.639215 3368 catalog_manager.cc:671] Loaded metadata for table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]
I20250626 14:04:56.639425 3368 catalog_manager.cc:671] Loaded metadata for table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]
I20250626 14:04:56.639648 3368 catalog_manager.cc:671] Loaded metadata for table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]
I20250626 14:04:56.639851 3368 catalog_manager.cc:671] Loaded metadata for table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]
I20250626 14:04:56.640048 3368 catalog_manager.cc:671] Loaded metadata for table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]
I20250626 14:04:56.640331 3368 catalog_manager.cc:671] Loaded metadata for table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]
I20250626 14:04:56.641521 3368 tablet_loader.cc:96] loaded metadata for tablet 025415f7a05746e19d5ed199246fc949 (table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a])
I20250626 14:04:56.641853 3368 tablet_loader.cc:96] loaded metadata for tablet 0db0b418e8b8412da3dc098ef6aab940 (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140])
I20250626 14:04:56.642107 3368 tablet_loader.cc:96] loaded metadata for tablet 1e307e729d7d42a68807263959c223a6 (table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb])
I20250626 14:04:56.642269 3368 tablet_loader.cc:96] loaded metadata for tablet 33d002d179974a95aaad3d84814d27f0 (table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75])
I20250626 14:04:56.642525 3368 tablet_loader.cc:96] loaded metadata for tablet 36de2c343f1345e28cb78616a9c25526 (table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233])
I20250626 14:04:56.643159 3368 tablet_loader.cc:96] loaded metadata for tablet 730ed7238006463eb08803a70330a43f (table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9])
I20250626 14:04:56.643499 3368 tablet_loader.cc:96] loaded metadata for tablet 94c7591e52cd4d88946f7e98f15e2b94 (table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447])
I20250626 14:04:56.643748 3368 tablet_loader.cc:96] loaded metadata for tablet a679d473653e403584d3271e7ce68515 (table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80])
I20250626 14:04:56.643905 3368 tablet_loader.cc:96] loaded metadata for tablet aa32cb56ec7346deac266cfb5e8279c5 (table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03])
I20250626 14:04:56.644150 3368 tablet_loader.cc:96] loaded metadata for tablet b25221169c9c48718276736fe657f378 (table default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843])
I20250626 14:04:56.644752 3368 tablet_loader.cc:96] loaded metadata for tablet f1c96426790242fdb8224137d55939f2 (table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c])
I20250626 14:04:56.645135 3368 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 14:04:56.645972 3368 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:56.646057 3368 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 14:04:56.647058 3368 catalog_manager.cc:1506] Loading token signing keys...
I20250626 14:04:56.647819 3368 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Loaded TSK: 0
I20250626 14:04:56.648739 3368 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 14:04:56.649482 3368 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250626 14:04:56.650192 3368 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 13
I20250626 14:04:56.657737 3295 master_service.cc:948] ReplaceTablet: received request to replace tablet 36de2c343f1345e28cb78616a9c25526 from {username='slave'} at 127.0.0.1:41672
I20250626 14:04:56.657835 3296 master_service.cc:948] ReplaceTablet: received request to replace tablet f1c96426790242fdb8224137d55939f2 from {username='slave'} at 127.0.0.1:41672
I20250626 14:04:56.658061 3294 master_service.cc:948] ReplaceTablet: received request to replace tablet aa32cb56ec7346deac266cfb5e8279c5 from {username='slave'} at 127.0.0.1:41672
I20250626 14:04:56.658231 3293 master_service.cc:948] ReplaceTablet: received request to replace tablet 36de2c343f1345e28cb78616a9c25526 from {username='slave'} at 127.0.0.1:41672
I20250626 14:04:56.658360 3292 master_service.cc:948] ReplaceTablet: received request to replace tablet 1e307e729d7d42a68807263959c223a6 from {username='slave'} at 127.0.0.1:41672
I20250626 14:04:56.658396 3289 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41672:
table { table_name: "default.Table_fb8ebc7aa8894f28b4d79252382f356a" } new_table_name: "default.Table_7d5a9b0c3ae440b89dac3ca5dd8e81ef" modify_external_catalogs: true
I20250626 14:04:56.659238 3288 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41672:
name: "default.Table_f98cc740eae3444b88bb688eb8c8a316"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:56.659600 3290 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41672:
table { table_name: "default.Table_e6e7d4c291af4d29baf43e204c5292f0" } new_table_name: "default.Table_f01c05221e5c4e75bf700b9474072881" modify_external_catalogs: true
I20250626 14:04:56.659710 3291 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41672:
table { table_name: "default.Table_0be8dc44c93040319885961d6e4a0608" } modify_external_catalogs: true
I20250626 14:04:56.659824 3287 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41672:
table { table_name: "default.Table_f9b5b5e2dcf64f9e889c618242274d25" } modify_external_catalogs: true
I20250626 14:04:56.659839 3291 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41672:
table { table_name: "default.Table_0be8dc44c93040319885961d6e4a0608" } modify_external_catalogs: true
I20250626 14:04:56.659904 3287 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41672:
table { table_name: "default.Table_f9b5b5e2dcf64f9e889c618242274d25" } modify_external_catalogs: true
I20250626 14:04:56.664041 3293 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41672:
name: "default.Table_5bf40f09daaa4240a8bcab30e1aa341f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:56.664408 3295 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Sending DeleteTablet for 3 replicas of tablet 36de2c343f1345e28cb78616a9c25526
I20250626 14:04:56.665740 3296 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Sending DeleteTablet for 3 replicas of tablet f1c96426790242fdb8224137d55939f2
I20250626 14:04:56.666010 3296 catalog_manager.cc:6652] ReplaceTablet: tablet f1c96426790242fdb8224137d55939f2 deleted and replaced by tablet c744a0827e7943ba9a6ac09cf5e0a9bd
I20250626 14:04:56.666195 3296 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41672:
table { table_name: "default.Table_3a3f21b3c0e54e438561e02ab79fa36b" } new_table_name: "default.Table_3633fc38c5374180a081fab63a1eed84" modify_external_catalogs: true
I20250626 14:04:56.666014 3295 catalog_manager.cc:6652] ReplaceTablet: tablet 36de2c343f1345e28cb78616a9c25526 deleted and replaced by tablet 7d559ff794ca4485bc682b6b8f5b8e0e
I20250626 14:04:56.666929 3295 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41672:
name: "default.Table_f7012cd4bf5d40d6af93cf9b01405dcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250626 14:04:56.667380 3350 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 16, previous 16; dumping first 4 out of 8 received events: NotificationEvent(eventId=14, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_72f9307d2f5247a0ae4499c953ea1650, message={"server":"","servicePrincipal":"","db":"default","table":"table_72f9307d2f5247a0ae4499c953ea1650","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_72f9307d2f5247a0ae4499c953ea1650\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_72f9307d2f5247a0ae4499c953ea1650\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_72f9307d2f5247a0ae4499c953ea1650\",\"kudu.table_id\":\"61be84e63b2e478b9da5619fdbdffee9\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=15, eventTime=1750946674, eventType=CREATE_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=ALTER_TABLE, dbName=default, tableName=table_7d5a9b0c3ae440b89dac3ca5dd8e81ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_fb8ebc7aa8894f28b4d79252382f356a","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_fb8ebc7aa8894f28b4d79252382f356a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_fb8ebc7aa8894f28b4d79252382f356a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fb8ebc7aa8894f28b4d79252382f356a\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250626 14:04:56.667730 3292 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Sending DeleteTablet for 3 replicas of tablet 1e307e729d7d42a68807263959c223a6
I20250626 14:04:56.668035 3292 catalog_manager.cc:6652] ReplaceTablet: tablet 1e307e729d7d42a68807263959c223a6 deleted and replaced by tablet 48adfabcc2d84b6a9bfb7134f2c860ce
I20250626 14:04:56.668238 3292 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41672:
table { table_name: "default.Table_42d07c2b16314b1999eea5f3878944fe" } modify_external_catalogs: true
I20250626 14:04:56.668363 3292 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41672:
table { table_name: "default.Table_42d07c2b16314b1999eea5f3878944fe" } modify_external_catalogs: true
@ 0x7f534f750dcd google::LogMessage::Fail() at ??:0
I20250626 14:04:56.673343 1787 tablet_service.cc:1515] Processing DeleteTablet for tablet f1c96426790242fdb8224137d55939f2 with delete_type TABLET_DATA_DELETED (replaced by tablet c744a0827e7943ba9a6ac09cf5e0a9bd) from {username='slave'} at 127.0.0.1:39436
I20250626 14:04:56.673604 1789 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e307e729d7d42a68807263959c223a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 48adfabcc2d84b6a9bfb7134f2c860ce) from {username='slave'} at 127.0.0.1:39436
I20250626 14:04:56.673362 1790 tablet_service.cc:1515] Processing DeleteTablet for tablet 36de2c343f1345e28cb78616a9c25526 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d559ff794ca4485bc682b6b8f5b8e0e) from {username='slave'} at 127.0.0.1:39436
I20250626 14:04:56.673691 1921 tablet_service.cc:1515] Processing DeleteTablet for tablet 36de2c343f1345e28cb78616a9c25526 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d559ff794ca4485bc682b6b8f5b8e0e) from {username='slave'} at 127.0.0.1:46212
I20250626 14:04:56.673955 1922 tablet_service.cc:1515] Processing DeleteTablet for tablet f1c96426790242fdb8224137d55939f2 with delete_type TABLET_DATA_DELETED (replaced by tablet c744a0827e7943ba9a6ac09cf5e0a9bd) from {username='slave'} at 127.0.0.1:46212
I20250626 14:04:56.674077 1920 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e307e729d7d42a68807263959c223a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 48adfabcc2d84b6a9bfb7134f2c860ce) from {username='slave'} at 127.0.0.1:46212
I20250626 14:04:56.674077 3383 tablet_replica.cc:331] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:56.667558 3294 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Sending DeleteTablet for 3 replicas of tablet aa32cb56ec7346deac266cfb5e8279c5
I20250626 14:04:56.674216 3383 raft_consensus.cc:2241] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus shutting down.
@ 0x7f534f754b93 google::LogMessage::SendToLog() at ??:0
I20250626 14:04:56.674327 3383 raft_consensus.cc:2270] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:56.674448 3294 catalog_manager.cc:6652] ReplaceTablet: tablet aa32cb56ec7346deac266cfb5e8279c5 deleted and replaced by tablet b85960777e004b88850a7c16df33a4fd
I20250626 14:04:56.674665 3294 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41672:
table { table_name: "default.Table_afd26acd01c7422fbb06520af5a4f76c" } new_table_name: "default.Table_4d81e86bd6614c3db77efeaca0daa56f" modify_external_catalogs: true
I20250626 14:04:56.675253 1920 tablet_service.cc:1515] Processing DeleteTablet for tablet aa32cb56ec7346deac266cfb5e8279c5 with delete_type TABLET_DATA_DELETED (replaced by tablet b85960777e004b88850a7c16df33a4fd) from {username='slave'} at 127.0.0.1:46212
I20250626 14:04:56.674722 3383 ts_tablet_manager.cc:1905] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:56.675532 1790 tablet_service.cc:1515] Processing DeleteTablet for tablet aa32cb56ec7346deac266cfb5e8279c5 with delete_type TABLET_DATA_DELETED (replaced by tablet b85960777e004b88850a7c16df33a4fd) from {username='slave'} at 127.0.0.1:39436
I20250626 14:04:56.676023 1657 tablet_service.cc:1515] Processing DeleteTablet for tablet f1c96426790242fdb8224137d55939f2 with delete_type TABLET_DATA_DELETED (replaced by tablet c744a0827e7943ba9a6ac09cf5e0a9bd) from {username='slave'} at 127.0.0.1:37088
I20250626 14:04:56.676029 1654 tablet_service.cc:1515] Processing DeleteTablet for tablet aa32cb56ec7346deac266cfb5e8279c5 with delete_type TABLET_DATA_DELETED (replaced by tablet b85960777e004b88850a7c16df33a4fd) from {username='slave'} at 127.0.0.1:37088
I20250626 14:04:56.676297 1656 tablet_service.cc:1515] Processing DeleteTablet for tablet 36de2c343f1345e28cb78616a9c25526 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d559ff794ca4485bc682b6b8f5b8e0e) from {username='slave'} at 127.0.0.1:37088
I20250626 14:04:56.676405 1658 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e307e729d7d42a68807263959c223a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 48adfabcc2d84b6a9bfb7134f2c860ce) from {username='slave'} at 127.0.0.1:37088
@ 0x7f534f7507cc google::LogMessage::Flush() at ??:0
I20250626 14:04:56.678149 3383 ts_tablet_manager.cc:1918] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:56.678221 3383 log.cc:1199] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/f1c96426790242fdb8224137d55939f2
I20250626 14:04:56.678510 3383 ts_tablet_manager.cc:1939] T f1c96426790242fdb8224137d55939f2 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:56.678933 3384 tablet_replica.cc:331] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:56.679037 3386 tablet_replica.cc:331] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:56.679091 3384 raft_consensus.cc:2241] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:56.679167 3386 raft_consensus.cc:2241] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:56.679270 3386 raft_consensus.cc:2270] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:56.679298 3384 raft_consensus.cc:2270] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f534f750a11 google::LogMessage::~LogMessage() at ??:0
I20250626 14:04:56.679648 3384 ts_tablet_manager.cc:1905] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:56.679648 3386 ts_tablet_manager.cc:1905] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:56.681762 3384 ts_tablet_manager.cc:1918] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:56.681777 3386 ts_tablet_manager.cc:1918] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:56.681830 3384 log.cc:1199] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/36de2c343f1345e28cb78616a9c25526
I20250626 14:04:56.681834 3386 log.cc:1199] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/f1c96426790242fdb8224137d55939f2
I20250626 14:04:56.682111 3386 ts_tablet_manager.cc:1939] T f1c96426790242fdb8224137d55939f2 P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:56.682111 3384 ts_tablet_manager.cc:1939] T 36de2c343f1345e28cb78616a9c25526 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:56.682775 1658 tablet_service.cc:1468] Processing CreateTablet for tablet c744a0827e7943ba9a6ac09cf5e0a9bd (DEFAULT_TABLE table=default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:56.682866 1920 tablet_service.cc:1468] Processing CreateTablet for tablet c744a0827e7943ba9a6ac09cf5e0a9bd (DEFAULT_TABLE table=default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:56.683032 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c744a0827e7943ba9a6ac09cf5e0a9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:56.682996 1922 tablet_service.cc:1468] Processing CreateTablet for tablet 7d559ff794ca4485bc682b6b8f5b8e0e (DEFAULT_TABLE table=default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:56.683108 1920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c744a0827e7943ba9a6ac09cf5e0a9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:56.683210 3383 tablet_replica.cc:331] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:56.683233 1922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d559ff794ca4485bc682b6b8f5b8e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:56.683344 3383 raft_consensus.cc:2241] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:56.683452 3383 raft_consensus.cc:2270] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:56.685845 3383 ts_tablet_manager.cc:1905] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:56.685926 3282 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet f1c96426790242fdb8224137d55939f2 (table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]) successfully deleted
I20250626 14:04:56.685912 1790 tablet_service.cc:1468] Processing CreateTablet for tablet c744a0827e7943ba9a6ac09cf5e0a9bd (DEFAULT_TABLE table=default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:56.686219 1790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c744a0827e7943ba9a6ac09cf5e0a9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:56.686842 3386 tablet_replica.cc:331] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:56.686995 3386 raft_consensus.cc:2241] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:56.687207 3386 raft_consensus.cc:2270] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:56.687651 3386 ts_tablet_manager.cc:1905] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:56.688807 1789 tablet_service.cc:1468] Processing CreateTablet for tablet 7d559ff794ca4485bc682b6b8f5b8e0e (DEFAULT_TABLE table=default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:56.689046 1789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d559ff794ca4485bc682b6b8f5b8e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:56.690092 3386 ts_tablet_manager.cc:1918] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:56.690160 3386 log.cc:1199] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/aa32cb56ec7346deac266cfb5e8279c5
I20250626 14:04:56.690183 3383 ts_tablet_manager.cc:1918] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:56.690258 3383 log.cc:1199] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/1e307e729d7d42a68807263959c223a6
I20250626 14:04:56.690420 3386 ts_tablet_manager.cc:1939] T aa32cb56ec7346deac266cfb5e8279c5 P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:56.690491 3383 ts_tablet_manager.cc:1939] T 1e307e729d7d42a68807263959c223a6 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:56.691116 3383 tablet_replica.cc:331] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:56.691241 3383 raft_consensus.cc:2241] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:56.691218 3282 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet 1e307e729d7d42a68807263959c223a6 (table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]) successfully deleted
I20250626 14:04:56.691247 1658 tablet_service.cc:1468] Processing CreateTablet for tablet 7d559ff794ca4485bc682b6b8f5b8e0e (DEFAULT_TABLE table=default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:56.691339 3383 raft_consensus.cc:2270] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:56.691668 3383 ts_tablet_manager.cc:1905] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:56.691808 3384 tablet_replica.cc:331] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:56.691922 3384 raft_consensus.cc:2241] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:56.691970 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d559ff794ca4485bc682b6b8f5b8e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:56.691948 3282 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet 36de2c343f1345e28cb78616a9c25526 (table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]) successfully deleted
I20250626 14:04:56.692131 3384 raft_consensus.cc:2270] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:56.692142 3283 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet f1c96426790242fdb8224137d55939f2 (table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]) successfully deleted
I20250626 14:04:56.692497 3384 ts_tablet_manager.cc:1905] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:56.693531 3383 ts_tablet_manager.cc:1918] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:56.693599 3383 log.cc:1199] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/36de2c343f1345e28cb78616a9c25526
I20250626 14:04:56.693852 3383 ts_tablet_manager.cc:1939] T 36de2c343f1345e28cb78616a9c25526 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:56.694363 3383 tablet_replica.cc:331] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:56.694451 3282 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet 36de2c343f1345e28cb78616a9c25526 (table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]) successfully deleted
I20250626 14:04:56.694499 3383 raft_consensus.cc:2241] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:56.694478 1922 tablet_service.cc:1468] Processing CreateTablet for tablet b85960777e004b88850a7c16df33a4fd (DEFAULT_TABLE table=default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:56.694599 3383 raft_consensus.cc:2270] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:56.694617 3384 ts_tablet_manager.cc:1918] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:56.694675 3384 log.cc:1199] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/f1c96426790242fdb8224137d55939f2
I20250626 14:04:56.694707 1922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85960777e004b88850a7c16df33a4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:56.694912 3384 ts_tablet_manager.cc:1939] T f1c96426790242fdb8224137d55939f2 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:56.694981 3383 ts_tablet_manager.cc:1905] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:56.695498 3384 tablet_replica.cc:331] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:56.695603 3384 raft_consensus.cc:2241] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:56.695623 3282 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet f1c96426790242fdb8224137d55939f2 (table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]) successfully deleted
I20250626 14:04:56.695780 3384 raft_consensus.cc:2270] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:56.696123 3384 ts_tablet_manager.cc:1905] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:56.696818 3383 ts_tablet_manager.cc:1918] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:56.696879 3383 log.cc:1199] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/aa32cb56ec7346deac266cfb5e8279c5
I20250626 14:04:56.697053 3386 tablet_replica.cc:331] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:56.697115 3383 ts_tablet_manager.cc:1939] T aa32cb56ec7346deac266cfb5e8279c5 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:56.697093 1789 tablet_service.cc:1468] Processing CreateTablet for tablet b85960777e004b88850a7c16df33a4fd (DEFAULT_TABLE table=default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:56.697185 3386 raft_consensus.cc:2241] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:56.697302 3386 raft_consensus.cc:2270] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:56.697407 1789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85960777e004b88850a7c16df33a4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:56.697623 3386 ts_tablet_manager.cc:1905] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f53546e9c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250626 14:04:56.699108 3386 ts_tablet_manager.cc:1918] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:56.699158 3386 log.cc:1199] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/36de2c343f1345e28cb78616a9c25526
I20250626 14:04:56.699342 3386 ts_tablet_manager.cc:1939] T 36de2c343f1345e28cb78616a9c25526 P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:56.699975 3386 tablet_replica.cc:331] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:56.700089 3386 raft_consensus.cc:2241] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:56.700177 3386 raft_consensus.cc:2270] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:56.700277 3282 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet aa32cb56ec7346deac266cfb5e8279c5 (table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]) successfully deleted
I20250626 14:04:56.700502 3386 ts_tablet_manager.cc:1905] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:56.700733 3389 tablet_bootstrap.cc:492] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:56.700989 3384 ts_tablet_manager.cc:1918] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:56.701025 3283 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet aa32cb56ec7346deac266cfb5e8279c5 (table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]) successfully deleted
I20250626 14:04:56.701046 1922 tablet_service.cc:1468] Processing CreateTablet for tablet 48adfabcc2d84b6a9bfb7134f2c860ce (DEFAULT_TABLE table=default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:56.701149 3283 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet 36de2c343f1345e28cb78616a9c25526 (table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]) successfully deleted
I20250626 14:04:56.701238 3388 tablet_bootstrap.cc:492] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:56.701318 1922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48adfabcc2d84b6a9bfb7134f2c860ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:56.701658 1656 tablet_service.cc:1468] Processing CreateTablet for tablet 48adfabcc2d84b6a9bfb7134f2c860ce (DEFAULT_TABLE table=default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:56.701884 1656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48adfabcc2d84b6a9bfb7134f2c860ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:56.702597 3386 ts_tablet_manager.cc:1918] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:56.702667 3386 log.cc:1199] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/1e307e729d7d42a68807263959c223a6
I20250626 14:04:56.703130 3386 ts_tablet_manager.cc:1939] T 1e307e729d7d42a68807263959c223a6 P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
@ 0x7f53546e8ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250626 14:04:56.704466 3389 tablet_bootstrap.cc:654] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:56.701051 3384 log.cc:1199] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/1e307e729d7d42a68807263959c223a6
I20250626 14:04:56.705346 3384 ts_tablet_manager.cc:1939] T 1e307e729d7d42a68807263959c223a6 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:56.706452 3384 tablet_replica.cc:331] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
@ 0x7f53546e830b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250626 14:04:56.706748 3384 raft_consensus.cc:2241] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250626 14:04:56.706815 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 21: this message will repeat every 5th retry.
I20250626 14:04:56.706856 3384 raft_consensus.cc:2270] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:56.707221 3384 ts_tablet_manager.cc:1905] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:56.707278 3282 catalog_manager.cc:4928] TS 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): tablet 1e307e729d7d42a68807263959c223a6 (table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]) successfully deleted
I20250626 14:04:56.707453 3283 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet 1e307e729d7d42a68807263959c223a6 (table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]) successfully deleted
I20250626 14:04:56.707830 1658 tablet_service.cc:1468] Processing CreateTablet for tablet b85960777e004b88850a7c16df33a4fd (DEFAULT_TABLE table=default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:56.708084 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85960777e004b88850a7c16df33a4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:56.709302 3384 ts_tablet_manager.cc:1918] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:56.709362 3384 log.cc:1199] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/aa32cb56ec7346deac266cfb5e8279c5
@ 0x7f53546eda12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250626 14:04:56.709463 1789 tablet_service.cc:1468] Processing CreateTablet for tablet 48adfabcc2d84b6a9bfb7134f2c860ce (DEFAULT_TABLE table=default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:56.709607 3384 ts_tablet_manager.cc:1939] T aa32cb56ec7346deac266cfb5e8279c5 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:56.709712 1789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48adfabcc2d84b6a9bfb7134f2c860ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:56.710131 3388 tablet_bootstrap.cc:654] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
@ 0x559af5dc8752 std::function<>::operator()() at ??:0
I20250626 14:04:56.711011 3387 tablet_bootstrap.cc:492] T c744a0827e7943ba9a6ac09cf5e0a9bd P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:56.711370 3389 tablet_bootstrap.cc:492] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:56.711447 3389 ts_tablet_manager.cc:1397] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250626 14:04:56.711962 3389 raft_consensus.cc:357] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.712091 3389 raft_consensus.cc:383] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:56.712131 3389 raft_consensus.cc:738] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:56.712270 3389 consensus_queue.cc:260] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.712409 3387 tablet_bootstrap.cc:654] T c744a0827e7943ba9a6ac09cf5e0a9bd P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:56.712606 3389 ts_tablet_manager.cc:1428] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 14:04:56.712751 3389 tablet_bootstrap.cc:492] T 7d559ff794ca4485bc682b6b8f5b8e0e P e517893e13f546c0b40d976633b49651: Bootstrap starting.
@ 0x7f5350ba20a1 kudu::Thread::SuperviseThread() at ??:0
I20250626 14:04:56.713716 3389 tablet_bootstrap.cc:654] T 7d559ff794ca4485bc682b6b8f5b8e0e P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:56.714951 3389 tablet_bootstrap.cc:492] T 7d559ff794ca4485bc682b6b8f5b8e0e P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:56.715032 3389 ts_tablet_manager.cc:1397] T 7d559ff794ca4485bc682b6b8f5b8e0e P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
@ 0x7f53513916db start_thread at ??:0
I20250626 14:04:56.715790 3389 raft_consensus.cc:357] T 7d559ff794ca4485bc682b6b8f5b8e0e P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.715901 3389 raft_consensus.cc:383] T 7d559ff794ca4485bc682b6b8f5b8e0e P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:56.715942 3389 raft_consensus.cc:738] T 7d559ff794ca4485bc682b6b8f5b8e0e P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:56.716070 3389 consensus_queue.cc:260] T 7d559ff794ca4485bc682b6b8f5b8e0e P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.716415 3389 ts_tablet_manager.cc:1428] T 7d559ff794ca4485bc682b6b8f5b8e0e P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:56.716722 3389 tablet_bootstrap.cc:492] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651: Bootstrap starting.
@ 0x7f534e90771f clone at ??:0
I20250626 14:04:56.717974 3389 tablet_bootstrap.cc:654] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:56.719296 3388 tablet_bootstrap.cc:492] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:56.719380 3388 ts_tablet_manager.cc:1397] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250626 14:04:56.719568 3387 tablet_bootstrap.cc:492] T c744a0827e7943ba9a6ac09cf5e0a9bd P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:56.719650 3387 ts_tablet_manager.cc:1397] T c744a0827e7943ba9a6ac09cf5e0a9bd P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20250626 14:04:56.719810 3389 tablet_bootstrap.cc:492] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:56.719890 3389 ts_tablet_manager.cc:1397] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:56.719873 3388 raft_consensus.cc:357] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.719969 3388 raft_consensus.cc:383] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:56.720000 3388 raft_consensus.cc:738] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:56.720101 3387 raft_consensus.cc:357] T c744a0827e7943ba9a6ac09cf5e0a9bd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.720199 3387 raft_consensus.cc:383] T c744a0827e7943ba9a6ac09cf5e0a9bd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:56.720165 3388 consensus_queue.cc:260] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.720248 3387 raft_consensus.cc:738] T c744a0827e7943ba9a6ac09cf5e0a9bd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:56.720553 3388 ts_tablet_manager.cc:1428] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:56.720373 3387 consensus_queue.cc:260] T c744a0827e7943ba9a6ac09cf5e0a9bd P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.720850 3387 ts_tablet_manager.cc:1428] T c744a0827e7943ba9a6ac09cf5e0a9bd P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:56.721076 3387 tablet_bootstrap.cc:492] T 7d559ff794ca4485bc682b6b8f5b8e0e P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:56.721237 3389 raft_consensus.cc:357] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.721335 3389 raft_consensus.cc:383] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:56.721370 3389 raft_consensus.cc:738] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:56.721477 3389 consensus_queue.cc:260] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.721769 3389 ts_tablet_manager.cc:1428] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 14:04:56.721920 3389 tablet_bootstrap.cc:492] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:56.722510 3387 tablet_bootstrap.cc:654] T 7d559ff794ca4485bc682b6b8f5b8e0e P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:56.723001 3389 tablet_bootstrap.cc:654] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:56.723215 3388 tablet_bootstrap.cc:492] T c744a0827e7943ba9a6ac09cf5e0a9bd P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:56.724021 3389 tablet_bootstrap.cc:492] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:56.724093 3389 ts_tablet_manager.cc:1397] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 14:04:56.724182 3388 tablet_bootstrap.cc:654] T c744a0827e7943ba9a6ac09cf5e0a9bd P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:56.724617 3389 raft_consensus.cc:357] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.724736 3389 raft_consensus.cc:383] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:56.724778 3389 raft_consensus.cc:738] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:56.724927 3389 consensus_queue.cc:260] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.725296 3389 ts_tablet_manager.cc:1428] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:56.726060 3388 tablet_bootstrap.cc:492] T c744a0827e7943ba9a6ac09cf5e0a9bd P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:56.726156 3388 ts_tablet_manager.cc:1397] T c744a0827e7943ba9a6ac09cf5e0a9bd P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:56.726703 3388 raft_consensus.cc:357] T c744a0827e7943ba9a6ac09cf5e0a9bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.726811 3388 raft_consensus.cc:383] T c744a0827e7943ba9a6ac09cf5e0a9bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:56.726842 3388 raft_consensus.cc:738] T c744a0827e7943ba9a6ac09cf5e0a9bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:56.727002 3388 consensus_queue.cc:260] T c744a0827e7943ba9a6ac09cf5e0a9bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.727221 3387 tablet_bootstrap.cc:492] T 7d559ff794ca4485bc682b6b8f5b8e0e P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:56.727304 3387 ts_tablet_manager.cc:1397] T 7d559ff794ca4485bc682b6b8f5b8e0e P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20250626 14:04:56.727361 3388 ts_tablet_manager.cc:1428] T c744a0827e7943ba9a6ac09cf5e0a9bd P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:56.727733 3388 tablet_bootstrap.cc:492] T b85960777e004b88850a7c16df33a4fd P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:56.727809 3387 raft_consensus.cc:357] T 7d559ff794ca4485bc682b6b8f5b8e0e P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.727919 3387 raft_consensus.cc:383] T 7d559ff794ca4485bc682b6b8f5b8e0e P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:56.727962 3387 raft_consensus.cc:738] T 7d559ff794ca4485bc682b6b8f5b8e0e P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:56.728082 3387 consensus_queue.cc:260] T 7d559ff794ca4485bc682b6b8f5b8e0e P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.728343 3387 ts_tablet_manager.cc:1428] T 7d559ff794ca4485bc682b6b8f5b8e0e P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:56.728519 3387 tablet_bootstrap.cc:492] T 48adfabcc2d84b6a9bfb7134f2c860ce P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:56.728806 3388 tablet_bootstrap.cc:654] T b85960777e004b88850a7c16df33a4fd P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:56.729555 3387 tablet_bootstrap.cc:654] T 48adfabcc2d84b6a9bfb7134f2c860ce P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:56.733109 3387 tablet_bootstrap.cc:492] T 48adfabcc2d84b6a9bfb7134f2c860ce P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:56.733170 3388 tablet_bootstrap.cc:492] T b85960777e004b88850a7c16df33a4fd P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:56.733188 3387 ts_tablet_manager.cc:1397] T 48adfabcc2d84b6a9bfb7134f2c860ce P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20250626 14:04:56.733258 3388 ts_tablet_manager.cc:1397] T b85960777e004b88850a7c16df33a4fd P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250626 14:04:56.733681 3387 raft_consensus.cc:357] T 48adfabcc2d84b6a9bfb7134f2c860ce P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.733793 3387 raft_consensus.cc:383] T 48adfabcc2d84b6a9bfb7134f2c860ce P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:56.733820 3387 raft_consensus.cc:738] T 48adfabcc2d84b6a9bfb7134f2c860ce P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:56.733932 3387 consensus_queue.cc:260] T 48adfabcc2d84b6a9bfb7134f2c860ce P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.734140 3388 raft_consensus.cc:357] T b85960777e004b88850a7c16df33a4fd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.734261 3388 raft_consensus.cc:383] T b85960777e004b88850a7c16df33a4fd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:56.734301 3388 raft_consensus.cc:738] T b85960777e004b88850a7c16df33a4fd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:56.734438 3388 consensus_queue.cc:260] T b85960777e004b88850a7c16df33a4fd P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.734660 3387 ts_tablet_manager.cc:1428] T 48adfabcc2d84b6a9bfb7134f2c860ce P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:56.734802 3388 ts_tablet_manager.cc:1428] T b85960777e004b88850a7c16df33a4fd P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 14:04:56.734851 3387 tablet_bootstrap.cc:492] T b85960777e004b88850a7c16df33a4fd P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:56.735003 3388 tablet_bootstrap.cc:492] T 48adfabcc2d84b6a9bfb7134f2c860ce P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:56.736208 3388 tablet_bootstrap.cc:654] T 48adfabcc2d84b6a9bfb7134f2c860ce P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:56.736824 3387 tablet_bootstrap.cc:654] T b85960777e004b88850a7c16df33a4fd P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:56.738607 3388 tablet_bootstrap.cc:492] T 48adfabcc2d84b6a9bfb7134f2c860ce P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:56.738688 3388 ts_tablet_manager.cc:1397] T 48adfabcc2d84b6a9bfb7134f2c860ce P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250626 14:04:56.739166 3388 raft_consensus.cc:357] T 48adfabcc2d84b6a9bfb7134f2c860ce P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.739264 3388 raft_consensus.cc:383] T 48adfabcc2d84b6a9bfb7134f2c860ce P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:56.739296 3388 raft_consensus.cc:738] T 48adfabcc2d84b6a9bfb7134f2c860ce P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:56.739410 3388 consensus_queue.cc:260] T 48adfabcc2d84b6a9bfb7134f2c860ce P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.739444 3387 tablet_bootstrap.cc:492] T b85960777e004b88850a7c16df33a4fd P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:56.739543 3387 ts_tablet_manager.cc:1397] T b85960777e004b88850a7c16df33a4fd P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20250626 14:04:56.739673 3388 ts_tablet_manager.cc:1428] T 48adfabcc2d84b6a9bfb7134f2c860ce P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:56.740048 3387 raft_consensus.cc:357] T b85960777e004b88850a7c16df33a4fd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.740151 3387 raft_consensus.cc:383] T b85960777e004b88850a7c16df33a4fd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:56.740191 3387 raft_consensus.cc:738] T b85960777e004b88850a7c16df33a4fd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:56.740307 3387 consensus_queue.cc:260] T b85960777e004b88850a7c16df33a4fd P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.740643 3387 ts_tablet_manager.cc:1428] T b85960777e004b88850a7c16df33a4fd P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:56.774287 2937 raft_consensus.cc:491] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:56.774454 2937 raft_consensus.cc:513] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.775010 2937 leader_election.cc:290] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:56.775490 1948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48adfabcc2d84b6a9bfb7134f2c860ce" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:56.775511 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48adfabcc2d84b6a9bfb7134f2c860ce" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:56.775650 1948 raft_consensus.cc:2466] T 48adfabcc2d84b6a9bfb7134f2c860ce P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:56.775650 1678 raft_consensus.cc:2466] T 48adfabcc2d84b6a9bfb7134f2c860ce P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:56.775987 1748 leader_election.cc:304] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:56.776203 2937 raft_consensus.cc:2802] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:56.776314 2937 raft_consensus.cc:491] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:56.776371 2937 raft_consensus.cc:3058] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:56.777335 2937 raft_consensus.cc:513] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.777673 2937 leader_election.cc:290] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:56.778038 1948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48adfabcc2d84b6a9bfb7134f2c860ce" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:56.778038 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48adfabcc2d84b6a9bfb7134f2c860ce" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:56.778165 1948 raft_consensus.cc:3058] T 48adfabcc2d84b6a9bfb7134f2c860ce P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:56.778165 1678 raft_consensus.cc:3058] T 48adfabcc2d84b6a9bfb7134f2c860ce P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:56.779155 1678 raft_consensus.cc:2466] T 48adfabcc2d84b6a9bfb7134f2c860ce P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:56.779155 1948 raft_consensus.cc:2466] T 48adfabcc2d84b6a9bfb7134f2c860ce P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:56.779474 1749 leader_election.cc:304] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:56.779680 2937 raft_consensus.cc:2802] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:56.779772 2937 raft_consensus.cc:695] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Becoming Leader. State: Replica: e517893e13f546c0b40d976633b49651, State: Running, Role: LEADER
I20250626 14:04:56.779927 2937 consensus_queue.cc:237] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.832934 2937 raft_consensus.cc:491] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:56.833139 2937 raft_consensus.cc:513] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.833775 2937 leader_election.cc:290] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:56.834043 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b85960777e004b88850a7c16df33a4fd" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:56.834174 1678 raft_consensus.cc:2466] T b85960777e004b88850a7c16df33a4fd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:56.834213 1948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b85960777e004b88850a7c16df33a4fd" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:56.834345 1948 raft_consensus.cc:2466] T b85960777e004b88850a7c16df33a4fd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:56.834543 1749 leader_election.cc:304] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:56.834758 2937 raft_consensus.cc:2802] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:56.834852 2937 raft_consensus.cc:491] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:56.834939 2937 raft_consensus.cc:3058] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:56.837690 2937 raft_consensus.cc:513] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.838410 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b85960777e004b88850a7c16df33a4fd" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:56.838430 1948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b85960777e004b88850a7c16df33a4fd" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:56.838551 1678 raft_consensus.cc:3058] T b85960777e004b88850a7c16df33a4fd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:56.838568 1948 raft_consensus.cc:3058] T b85960777e004b88850a7c16df33a4fd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:56.839520 1678 raft_consensus.cc:2466] T b85960777e004b88850a7c16df33a4fd P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:56.839615 1948 raft_consensus.cc:2466] T b85960777e004b88850a7c16df33a4fd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:56.839920 1749 leader_election.cc:304] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:56.840071 2937 leader_election.cc:290] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:56.840093 2794 raft_consensus.cc:2802] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:56.840163 2794 raft_consensus.cc:695] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Becoming Leader. State: Replica: e517893e13f546c0b40d976633b49651, State: Running, Role: LEADER
I20250626 14:04:56.840291 2794 consensus_queue.cc:237] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
W20250626 14:04:56.894974 1613 connection.cc:537] client connection to 127.1.11.253:44447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:56.895136 1883 connection.cc:537] server connection from 127.0.0.1:46212 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:56.895283 1883 connection.cc:537] client connection to 127.1.11.253:44447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:56.895380 2003 connection.cc:537] client connection to 127.1.11.253:44447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:56.895818 2003 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.11.253:44447)
W20250626 14:04:56.896260 2003 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.11.253:44447)
W20250626 14:04:56.896378 2003 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.11.253:44447)
W20250626 14:04:56.896479 3355 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 14:04:56.895811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.11.253:44447 (local address 127.1.11.195:52227)
0626 14:04:56.896009 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0626 14:04:56.896023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0626 14:04:56.896105 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0626 14:04:56.896363 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":118}
W20250626 14:04:56.897089 2003 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.11.253:44447)
W20250626 14:04:56.897117 3357 negotiation.cc:337] Failed RPC negotiation. Trace:
0626 14:04:56.895690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.11.253:44447 (local address 127.1.11.193:43221)
0626 14:04:56.895905 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0626 14:04:56.895921 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0626 14:04:56.896014 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0626 14:04:56.897011 (+ 997us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":117}
I20250626 14:04:56.965621 2794 raft_consensus.cc:491] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:56.965777 2794 raft_consensus.cc:513] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.966249 2794 leader_election.cc:290] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:56.966708 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c744a0827e7943ba9a6ac09cf5e0a9bd" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:56.966727 1948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c744a0827e7943ba9a6ac09cf5e0a9bd" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:56.966846 1678 raft_consensus.cc:2466] T c744a0827e7943ba9a6ac09cf5e0a9bd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:56.966861 1948 raft_consensus.cc:2466] T c744a0827e7943ba9a6ac09cf5e0a9bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 0.
I20250626 14:04:56.967156 1749 leader_election.cc:304] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:56.967339 2794 raft_consensus.cc:2802] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:56.967398 2794 raft_consensus.cc:491] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:56.967442 2794 raft_consensus.cc:3058] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:56.968418 2794 raft_consensus.cc:513] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.968775 2794 leader_election.cc:290] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:56.969101 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c744a0827e7943ba9a6ac09cf5e0a9bd" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:56.969225 1678 raft_consensus.cc:3058] T c744a0827e7943ba9a6ac09cf5e0a9bd P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:56.970006 1678 raft_consensus.cc:2466] T c744a0827e7943ba9a6ac09cf5e0a9bd P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
I20250626 14:04:56.970252 1948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c744a0827e7943ba9a6ac09cf5e0a9bd" candidate_uuid: "e517893e13f546c0b40d976633b49651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:56.970341 1749 leader_election.cc:304] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:56.970417 1948 raft_consensus.cc:3058] T c744a0827e7943ba9a6ac09cf5e0a9bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:56.970520 2794 raft_consensus.cc:2802] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:56.970616 2794 raft_consensus.cc:695] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Becoming Leader. State: Replica: e517893e13f546c0b40d976633b49651, State: Running, Role: LEADER
I20250626 14:04:56.970772 2794 consensus_queue.cc:237] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:56.971480 1948 raft_consensus.cc:2466] T c744a0827e7943ba9a6ac09cf5e0a9bd P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e517893e13f546c0b40d976633b49651 in term 1.
W20250626 14:04:57.041785 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 21: this message will repeat every 5th retry.
I20250626 14:04:57.059517 2906 raft_consensus.cc:491] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:57.059664 2906 raft_consensus.cc:513] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:57.060159 2906 leader_election.cc:290] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:57.060642 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d559ff794ca4485bc682b6b8f5b8e0e" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:57.060658 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d559ff794ca4485bc682b6b8f5b8e0e" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:57.060801 1678 raft_consensus.cc:2466] T 7d559ff794ca4485bc682b6b8f5b8e0e P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:57.060796 1813 raft_consensus.cc:2466] T 7d559ff794ca4485bc682b6b8f5b8e0e P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:57.061139 1884 leader_election.cc:304] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:57.061352 2906 raft_consensus.cc:2802] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:57.061399 2906 raft_consensus.cc:491] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:57.061446 2906 raft_consensus.cc:3058] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:57.062291 2906 raft_consensus.cc:513] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:57.062633 2906 leader_election.cc:290] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Requested vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:57.063301 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d559ff794ca4485bc682b6b8f5b8e0e" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:57.063448 1678 raft_consensus.cc:3058] T 7d559ff794ca4485bc682b6b8f5b8e0e P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:57.064502 1678 raft_consensus.cc:2466] T 7d559ff794ca4485bc682b6b8f5b8e0e P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:57.064680 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d559ff794ca4485bc682b6b8f5b8e0e" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:57.064826 1813 raft_consensus.cc:3058] T 7d559ff794ca4485bc682b6b8f5b8e0e P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:57.064860 1884 leader_election.cc:304] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:57.065059 2906 raft_consensus.cc:2802] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:57.065289 2906 raft_consensus.cc:695] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Running, Role: LEADER
I20250626 14:04:57.065434 2906 consensus_queue.cc:237] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:57.065920 1813 raft_consensus.cc:2466] T 7d559ff794ca4485bc682b6b8f5b8e0e P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:57.193821 3421 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 07532b7648e4414789ed2b099bad2b19)
I20250626 14:04:57.193939 3421 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:57.194377 3421 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
W20250626 14:04:57.195437 2964 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
W20250626 14:04:57.195560 2966 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:57.195621 2966 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:57.195741 3421 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 14:04:57.197796 2938 consensus_queue.cc:1035] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:57.201736 2937 consensus_queue.cc:1035] T 48adfabcc2d84b6a9bfb7134f2c860ce P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:57.259959 2937 consensus_queue.cc:1035] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:57.264159 2937 consensus_queue.cc:1035] T b85960777e004b88850a7c16df33a4fd P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 14:04:57.383885 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 21: this message will repeat every 5th retry.
I20250626 14:04:57.405570 3432 consensus_queue.cc:1035] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:57.411229 2938 consensus_queue.cc:1035] T c744a0827e7943ba9a6ac09cf5e0a9bd P e517893e13f546c0b40d976633b49651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:57.517995 2906 consensus_queue.cc:1035] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:57.521713 3419 consensus_queue.cc:1035] T 7d559ff794ca4485bc682b6b8f5b8e0e P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:04:57.741001 3440 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 07532b7648e4414789ed2b099bad2b19)
I20250626 14:04:57.741118 3440 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:57.741514 3440 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
W20250626 14:04:57.742522 2964 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
W20250626 14:04:57.742607 2966 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:57.742689 2966 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:57.742816 3440 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 14:04:58.632795 3450 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 07532b7648e4414789ed2b099bad2b19)
I20250626 14:04:58.632922 3450 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:58.633406 3450 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
W20250626 14:04:58.634804 2964 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
W20250626 14:04:58.634902 2966 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:04:58.634961 2966 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81db61eed5584e7e8b0cf40e5441f88e; no voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:58.635085 3450 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 14:04:58.699419 2948 master_runner.cc:186] Error getting master registration for 127.1.11.254:36707: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111),
I20250626 14:04:58.699564 2948 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:04:58.918095 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.253:44447
--webserver_interface=127.1.11.253
--webserver_port=37937
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250626 14:04:59.029326 3454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:04:59.029551 3454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:04:59.029592 3454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:04:59.032764 3454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:04:59.032829 3454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:04:59.032850 3454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:04:59.032867 3454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:04:59.036620 3454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.253:44447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.11.253
--webserver_port=37937
--never_fsync=true
--heap_profile_path=/tmp/kudu.3454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:04:59.037739 3454 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:04:59.038873 3454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:04:59.045647 3462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:59.045778 3454 server_base.cc:1048] running on GCE node
W20250626 14:04:59.045650 3460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:04:59.045660 3459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:04:59.046463 3454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:04:59.047305 3454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:04:59.048460 3454 hybrid_clock.cc:648] HybridClock initialized: now 1750946699048451 us; error 49 us; skew 500 ppm
I20250626 14:04:59.050323 3454 webserver.cc:469] Webserver started at http://127.1.11.253:37937/ using document root <none> and password file <none>
I20250626 14:04:59.050869 3454 fs_manager.cc:362] Metadata directory not provided
I20250626 14:04:59.050923 3454 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:04:59.054476 3454 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20250626 14:04:59.056622 3468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:04:59.057627 3454 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250626 14:04:59.057741 3454 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
uuid: "07532b7648e4414789ed2b099bad2b19"
format_stamp: "Formatted at 2025-06-26 14:04:30 on dist-test-slave-rptj"
I20250626 14:04:59.058127 3454 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:04:59.083706 3454 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:04:59.084393 3454 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:04:59.084605 3454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:04:59.091658 3454 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.253:44447
I20250626 14:04:59.091670 3520 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.253:44447 every 8 connection(s)
I20250626 14:04:59.092845 3454 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-1/data/info.pb
I20250626 14:04:59.094246 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 3454
I20250626 14:04:59.097322 3521 sys_catalog.cc:263] Verifying existing consensus state
I20250626 14:04:59.099085 3521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Bootstrap starting.
I20250626 14:04:59.109619 1726 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:59.113039 3521 log.cc:826] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Log is configured to *not* fsync() on all Append() calls
I20250626 14:04:59.113715 1996 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:59.154814 3521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Bootstrap replayed 1/1 log segments. Stats: ops{read=85 overwritten=0 applied=85 ignored=0} inserts{seen=57 ignored=0} mutations{seen=78 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 14:04:59.155228 3521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Bootstrap complete.
I20250626 14:04:59.157898 3521 raft_consensus.cc:357] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:59.158260 3521 raft_consensus.cc:738] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07532b7648e4414789ed2b099bad2b19, State: Initialized, Role: FOLLOWER
I20250626 14:04:59.158723 3521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 85, Last appended: 7.85, Last appended by leader: 85, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:59.159158 3528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:59.159276 3528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:59.159797 3521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:04:59.164399 3454 master_runner.cc:186] Error getting master registration for 127.1.11.254:36707: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111),
I20250626 14:04:59.166380 3542 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:59.166427 3542 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: loading cluster ID for follower catalog manager: success
I20250626 14:04:59.166471 3454 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:04:59.169929 3542 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: acquiring CA information for follower catalog manager: success
I20250626 14:04:59.172235 3542 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 14:04:59.174693 1861 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:59.178148 3485 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e517893e13f546c0b40d976633b49651" instance_seqno: 1750946671821196) as {username='slave'} at 127.1.11.194:34455; Asking this server to re-register.
I20250626 14:04:59.178856 1861 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:59.180493 3485 ts_manager.cc:194] Registered new tserver with Master: e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:59.448661 1726 heartbeater.cc:344] Connected to a master server at 127.1.11.253:44447
I20250626 14:04:59.449928 3485 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" instance_seqno: 1750946671637378) as {username='slave'} at 127.1.11.193:55291; Asking this server to re-register.
I20250626 14:04:59.450309 1726 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:59.451156 3485 ts_manager.cc:194] Registered new tserver with Master: 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:59.569828 3546 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 07532b7648e4414789ed2b099bad2b19)
I20250626 14:04:59.569970 3546 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:59.570464 3546 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
W20250626 14:04:59.571069 2964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250626 14:04:59.572002 2964 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
I20250626 14:04:59.574304 3495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81db61eed5584e7e8b0cf40e5441f88e" candidate_term: 8 candidate_status { last_received { term: 7 index: 85 } } ignore_live_leader: false dest_uuid: "07532b7648e4414789ed2b099bad2b19" is_pre_election: true
I20250626 14:04:59.574656 3495 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81db61eed5584e7e8b0cf40e5441f88e in term 7.
I20250626 14:04:59.574977 2966 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e; no voters: 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:59.575172 3546 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 14:04:59.575235 3546 raft_consensus.cc:491] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 07532b7648e4414789ed2b099bad2b19)
I20250626 14:04:59.575292 3546 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 7 FOLLOWER]: Advancing to term 8
I20250626 14:04:59.576431 3546 raft_consensus.cc:513] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:59.577163 3546 leader_election.cc:290] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 election: Requested vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447)
I20250626 14:04:59.577507 3495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81db61eed5584e7e8b0cf40e5441f88e" candidate_term: 8 candidate_status { last_received { term: 7 index: 85 } } ignore_live_leader: false dest_uuid: "07532b7648e4414789ed2b099bad2b19"
I20250626 14:04:59.577634 3495 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 7 FOLLOWER]: Advancing to term 8
W20250626 14:04:59.577790 2964 leader_election.cc:336] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
I20250626 14:04:59.579524 3495 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81db61eed5584e7e8b0cf40e5441f88e in term 8.
I20250626 14:04:59.579821 2966 leader_election.cc:304] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e; no voters: 24ad911d71b6433f8c7939a1d1f7b782
I20250626 14:04:59.579988 3546 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 8 FOLLOWER]: Leader election won for term 8
I20250626 14:04:59.580188 3546 raft_consensus.cc:695] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [term 8 LEADER]: Becoming Leader. State: Replica: 81db61eed5584e7e8b0cf40e5441f88e, State: Running, Role: LEADER
I20250626 14:04:59.580482 3546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 7.85, Last appended by leader: 85, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:04:59.581602 3550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81db61eed5584e7e8b0cf40e5441f88e. Latest consensus state: current_term: 8 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:59.581722 3550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:59.582020 3552 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 14:04:59.583730 3552 catalog_manager.cc:671] Loaded metadata for table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]
I20250626 14:04:59.584029 3552 catalog_manager.cc:671] Loaded metadata for table default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843]
I20250626 14:04:59.584425 3552 catalog_manager.cc:671] Loaded metadata for table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]
I20250626 14:04:59.584795 3552 catalog_manager.cc:671] Loaded metadata for table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]
I20250626 14:04:59.585050 3552 catalog_manager.cc:671] Loaded metadata for table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9]
I20250626 14:04:59.585265 3552 catalog_manager.cc:671] Loaded metadata for table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]
I20250626 14:04:59.585472 3552 catalog_manager.cc:671] Loaded metadata for table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]
I20250626 14:04:59.585682 3552 catalog_manager.cc:671] Loaded metadata for table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]
I20250626 14:04:59.585882 3552 catalog_manager.cc:671] Loaded metadata for table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]
I20250626 14:04:59.586107 3552 catalog_manager.cc:671] Loaded metadata for table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]
I20250626 14:04:59.586490 3552 catalog_manager.cc:671] Loaded metadata for table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]
I20250626 14:04:59.587699 3552 tablet_loader.cc:96] loaded metadata for tablet 025415f7a05746e19d5ed199246fc949 (table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a])
I20250626 14:04:59.588016 3552 tablet_loader.cc:96] loaded metadata for tablet 0db0b418e8b8412da3dc098ef6aab940 (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140])
I20250626 14:04:59.588375 3552 tablet_loader.cc:96] loaded metadata for tablet 33d002d179974a95aaad3d84814d27f0 (table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75])
I20250626 14:04:59.588819 3552 tablet_loader.cc:96] loaded metadata for tablet 48adfabcc2d84b6a9bfb7134f2c860ce (table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb])
I20250626 14:04:59.589654 3552 tablet_loader.cc:96] loaded metadata for tablet 730ed7238006463eb08803a70330a43f (table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9])
I20250626 14:04:59.589926 3552 tablet_loader.cc:96] loaded metadata for tablet 7d559ff794ca4485bc682b6b8f5b8e0e (table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233])
I20250626 14:04:59.590142 3552 tablet_loader.cc:96] loaded metadata for tablet 94c7591e52cd4d88946f7e98f15e2b94 (table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447])
I20250626 14:04:59.590359 3552 tablet_loader.cc:96] loaded metadata for tablet a679d473653e403584d3271e7ce68515 (table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80])
I20250626 14:04:59.590644 3552 tablet_loader.cc:96] loaded metadata for tablet b25221169c9c48718276736fe657f378 (table default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843])
I20250626 14:04:59.590863 3552 tablet_loader.cc:96] loaded metadata for tablet b85960777e004b88850a7c16df33a4fd (table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03])
I20250626 14:04:59.591027 3552 tablet_loader.cc:96] loaded metadata for tablet c744a0827e7943ba9a6ac09cf5e0a9bd (table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c])
I20250626 14:04:59.591799 3552 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 14:04:59.592485 3552 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:04:59.592561 3552 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 14:04:59.593356 3552 catalog_manager.cc:1506] Loading token signing keys...
I20250626 14:04:59.593941 3552 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Loaded TSK: 0
I20250626 14:04:59.594698 3552 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 14:04:59.595306 3552 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250626 14:04:59.596016 3552 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 13
I20250626 14:04:59.616302 3485 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" instance_seqno: 1750946672002466) as {username='slave'} at 127.1.11.195:57483; Asking this server to re-register.
I20250626 14:04:59.616792 1996 heartbeater.cc:461] Registering TS with master...
I20250626 14:04:59.617712 3485 ts_manager.cc:194] Registered new tserver with Master: 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:04:59.655797 2978 master_service.cc:948] ReplaceTablet: received request to replace tablet a679d473653e403584d3271e7ce68515 from {username='slave'} at 127.0.0.1:48894
I20250626 14:04:59.656296 2972 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48894:
name: "default.Table_8c238e2636e9491f938a48294f961982"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:59.656701 2975 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48894:
table { table_name: "default.Table_0b500f44e16e45e589c01ab7ad9a621b" } modify_external_catalogs: true
I20250626 14:04:59.656832 2975 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48894:
table { table_name: "default.Table_0b500f44e16e45e589c01ab7ad9a621b" } modify_external_catalogs: true
I20250626 14:04:59.657025 2977 master_service.cc:948] ReplaceTablet: received request to replace tablet a679d473653e403584d3271e7ce68515 from {username='slave'} at 127.0.0.1:48894
I20250626 14:04:59.657164 2979 master_service.cc:948] ReplaceTablet: received request to replace tablet 0db0b418e8b8412da3dc098ef6aab940 from {username='slave'} at 127.0.0.1:48894
I20250626 14:04:59.657445 2971 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48894:
name: "default.Table_213d95d0e97546c298d94c45091826cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:59.657840 2973 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48894:
name: "default.Table_33ee3521f64d4cefb8856259510e5f0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:59.657845 2980 master_service.cc:948] ReplaceTablet: received request to replace tablet 0db0b418e8b8412da3dc098ef6aab940 from {username='slave'} at 127.0.0.1:48894
I20250626 14:04:59.658289 2976 master_service.cc:948] ReplaceTablet: received request to replace tablet 0db0b418e8b8412da3dc098ef6aab940 from {username='slave'} at 127.0.0.1:48894
I20250626 14:04:59.660642 2974 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48894:
name: "default.Table_2c8dee0163784a60a8468b53619247f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:59.661358 3495 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Refusing update from remote peer 81db61eed5584e7e8b0cf40e5441f88e: Log matching property violated. Preceding OpId in replica: term: 7 index: 85. Preceding OpId from leader: term: 8 index: 88. (index mismatch)
I20250626 14:04:59.662145 3550 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [LEADER]: Connected to new peer: Peer: permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
F20250626 14:04:59.663602 3034 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 16, previous 16; dumping first 4 out of 8 received events: NotificationEvent(eventId=14, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_72f9307d2f5247a0ae4499c953ea1650, message={"server":"","servicePrincipal":"","db":"default","table":"table_72f9307d2f5247a0ae4499c953ea1650","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_72f9307d2f5247a0ae4499c953ea1650\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_72f9307d2f5247a0ae4499c953ea1650\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_72f9307d2f5247a0ae4499c953ea1650\",\"kudu.table_id\":\"61be84e63b2e478b9da5619fdbdffee9\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=15, eventTime=1750946674, eventType=CREATE_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=ALTER_TABLE, dbName=default, tableName=table_7d5a9b0c3ae440b89dac3ca5dd8e81ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_fb8ebc7aa8894f28b4d79252382f356a","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_fb8ebc7aa8894f28b4d79252382f356a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_fb8ebc7aa8894f28b4d79252382f356a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fb8ebc7aa8894f28b4d79252382f356a\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
W20250626 14:04:59.663626 2964 consensus_peers.cc:487] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e -> Peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Couldn't send request to peer 24ad911d71b6433f8c7939a1d1f7b782. Status: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 14:04:59.667006 3557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81db61eed5584e7e8b0cf40e5441f88e. Latest consensus state: current_term: 8 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
@ 0x7f5199f69dcd google::LogMessage::Fail() at ??:0
I20250626 14:04:59.667245 3557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:04:59.667557 1789 tablet_service.cc:1515] Processing DeleteTablet for tablet a4e9c6ce944b4ec6af44f7852cac93d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 33d002d179974a95aaad3d84814d27f0) from {username='slave'} at 127.0.0.1:39448
I20250626 14:04:59.667974 3561 tablet_replica.cc:331] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:59.668157 3561 raft_consensus.cc:2241] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:59.668686 3561 raft_consensus.cc:2270] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:59.668931 3554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:04:59.669039 3554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e [sys.catalog]: This master's current role is: LEADER
I20250626 14:04:59.669276 3561 ts_tablet_manager.cc:1905] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:59.669487 3557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "81db61eed5584e7e8b0cf40e5441f88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
@ 0x7f5199f6db93 google::LogMessage::SendToLog() at ??:0
I20250626 14:04:59.669602 3557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f5199f697cc google::LogMessage::Flush() at ??:0
I20250626 14:04:59.672127 3561 ts_tablet_manager.cc:1918] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:59.672204 3561 log.cc:1199] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:04:59.672500 3561 ts_tablet_manager.cc:1939] T a4e9c6ce944b4ec6af44f7852cac93d3 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:59.673411 2966 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet a4e9c6ce944b4ec6af44f7852cac93d3 (table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]) successfully deleted
@ 0x7f5199f69a11 google::LogMessage::~LogMessage() at ??:0
I20250626 14:04:59.674047 2978 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Sending DeleteTablet for 3 replicas of tablet a679d473653e403584d3271e7ce68515
I20250626 14:04:59.674520 2979 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 81db61eed5584e7e8b0cf40e5441f88e: Sending DeleteTablet for 3 replicas of tablet 0db0b418e8b8412da3dc098ef6aab940
I20250626 14:04:59.674772 2978 catalog_manager.cc:6652] ReplaceTablet: tablet a679d473653e403584d3271e7ce68515 deleted and replaced by tablet 199b89801c1a4cea9ef97d73f230337b
I20250626 14:04:59.674818 2977 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48894:
name: "default.Table_fdd6a46b24984290ac6f503f5684b003"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:59.674919 2979 catalog_manager.cc:6652] ReplaceTablet: tablet 0db0b418e8b8412da3dc098ef6aab940 deleted and replaced by tablet 7194d0f54a034d0da4f3a80eff67c28c
I20250626 14:04:59.674939 2978 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48894:
name: "default.Table_566f55c5d8c34043a37041fef8cb9f45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:59.675189 2979 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48894:
table { table_name: "default.Table_22f5000aea364833a97948c4478f7e27" } new_table_name: "default.Table_8268b369fb20436fb4e562f6a97eee1d" modify_external_catalogs: true
I20250626 14:04:59.676381 1790 tablet_service.cc:1515] Processing DeleteTablet for tablet 0db0b418e8b8412da3dc098ef6aab940 with delete_type TABLET_DATA_DELETED (replaced by tablet 7194d0f54a034d0da4f3a80eff67c28c) from {username='slave'} at 127.0.0.1:39448
I20250626 14:04:59.676447 2976 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48894:
table { table_name: "default.Table_0be8dc44c93040319885961d6e4a0608" } modify_external_catalogs: true
I20250626 14:04:59.676589 3561 tablet_replica.cc:331] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:59.676580 2976 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48894:
table { table_name: "default.Table_0be8dc44c93040319885961d6e4a0608" } modify_external_catalogs: true
I20250626 14:04:59.676728 3561 raft_consensus.cc:2241] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:04:59.676946 3561 raft_consensus.cc:2270] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:59.676950 1789 tablet_service.cc:1515] Processing DeleteTablet for tablet a679d473653e403584d3271e7ce68515 with delete_type TABLET_DATA_DELETED (replaced by tablet 199b89801c1a4cea9ef97d73f230337b) from {username='slave'} at 127.0.0.1:39448
I20250626 14:04:59.677263 2980 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48894:
name: "default.Table_f98cc740eae3444b88bb688eb8c8a316"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:04:59.677366 3561 ts_tablet_manager.cc:1905] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:59.680804 3561 ts_tablet_manager.cc:1918] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:59.680877 3561 log.cc:1199] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/0db0b418e8b8412da3dc098ef6aab940
I20250626 14:04:59.681121 3561 ts_tablet_manager.cc:1939] T 0db0b418e8b8412da3dc098ef6aab940 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:59.681779 2966 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet 0db0b418e8b8412da3dc098ef6aab940 (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]) successfully deleted
I20250626 14:04:59.682008 3561 tablet_replica.cc:331] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651: stopping tablet replica
I20250626 14:04:59.682129 3561 raft_consensus.cc:2241] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:59.682230 3561 raft_consensus.cc:2270] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:59.683032 1789 tablet_service.cc:1468] Processing CreateTablet for tablet 7194d0f54a034d0da4f3a80eff67c28c (DEFAULT_TABLE table=default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:59.683297 1789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7194d0f54a034d0da4f3a80eff67c28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:59.683816 1658 tablet_service.cc:1515] Processing DeleteTablet for tablet a679d473653e403584d3271e7ce68515 with delete_type TABLET_DATA_DELETED (replaced by tablet 199b89801c1a4cea9ef97d73f230337b) from {username='slave'} at 127.0.0.1:37094
I20250626 14:04:59.684095 1657 tablet_service.cc:1468] Processing CreateTablet for tablet 7194d0f54a034d0da4f3a80eff67c28c (DEFAULT_TABLE table=default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:59.684372 1657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7194d0f54a034d0da4f3a80eff67c28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:59.684445 3570 tablet_replica.cc:331] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:59.684580 3570 raft_consensus.cc:2241] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:59.684690 3570 raft_consensus.cc:2270] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:59.685035 3570 ts_tablet_manager.cc:1905] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:59.685376 3571 tablet_bootstrap.cc:492] T 7194d0f54a034d0da4f3a80eff67c28c P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:59.686295 1922 tablet_service.cc:1515] Processing DeleteTablet for tablet a679d473653e403584d3271e7ce68515 with delete_type TABLET_DATA_DELETED (replaced by tablet 199b89801c1a4cea9ef97d73f230337b) from {username='slave'} at 127.0.0.1:46226
I20250626 14:04:59.686461 3571 tablet_bootstrap.cc:654] T 7194d0f54a034d0da4f3a80eff67c28c P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:59.686614 1920 tablet_service.cc:1515] Processing DeleteTablet for tablet 0db0b418e8b8412da3dc098ef6aab940 with delete_type TABLET_DATA_DELETED (replaced by tablet 7194d0f54a034d0da4f3a80eff67c28c) from {username='slave'} at 127.0.0.1:46226
I20250626 14:04:59.686640 1656 tablet_service.cc:1515] Processing DeleteTablet for tablet 0db0b418e8b8412da3dc098ef6aab940 with delete_type TABLET_DATA_DELETED (replaced by tablet 7194d0f54a034d0da4f3a80eff67c28c) from {username='slave'} at 127.0.0.1:37094
I20250626 14:04:59.686868 1921 tablet_service.cc:1468] Processing CreateTablet for tablet 7194d0f54a034d0da4f3a80eff67c28c (DEFAULT_TABLE table=default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:59.686882 1654 tablet_service.cc:1468] Processing CreateTablet for tablet 199b89801c1a4cea9ef97d73f230337b (DEFAULT_TABLE table=default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:59.687098 1654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 199b89801c1a4cea9ef97d73f230337b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:59.687124 1921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7194d0f54a034d0da4f3a80eff67c28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:59.687223 3570 ts_tablet_manager.cc:1918] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:59.687280 3570 log.cc:1199] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/a679d473653e403584d3271e7ce68515
I20250626 14:04:59.687546 3570 ts_tablet_manager.cc:1939] T a679d473653e403584d3271e7ce68515 P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:59.688076 3570 tablet_replica.cc:331] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b: stopping tablet replica
I20250626 14:04:59.688269 3570 raft_consensus.cc:2241] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:59.688361 3570 raft_consensus.cc:2270] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:59.688719 3570 ts_tablet_manager.cc:1905] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:59.688838 1918 tablet_service.cc:1468] Processing CreateTablet for tablet 199b89801c1a4cea9ef97d73f230337b (DEFAULT_TABLE table=default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:59.689064 1918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 199b89801c1a4cea9ef97d73f230337b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:59.689263 2967 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet a679d473653e403584d3271e7ce68515 (table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]) successfully deleted
@ 0x7f519ef02c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250626 14:04:59.691088 1789 tablet_service.cc:1468] Processing CreateTablet for tablet 199b89801c1a4cea9ef97d73f230337b (DEFAULT_TABLE table=default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:04:59.691275 1789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 199b89801c1a4cea9ef97d73f230337b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:04:59.691385 3570 ts_tablet_manager.cc:1918] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:59.691454 3570 log.cc:1199] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-0/wal/wals/0db0b418e8b8412da3dc098ef6aab940
I20250626 14:04:59.691726 3570 ts_tablet_manager.cc:1939] T 0db0b418e8b8412da3dc098ef6aab940 P 4e57adb83e9f451488ed1d0034043e7b: Deleting consensus metadata
I20250626 14:04:59.692044 3575 tablet_bootstrap.cc:492] T 7194d0f54a034d0da4f3a80eff67c28c P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:59.692580 2967 catalog_manager.cc:4928] TS 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671): tablet 0db0b418e8b8412da3dc098ef6aab940 (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]) successfully deleted
I20250626 14:04:59.692721 3573 tablet_bootstrap.cc:492] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
I20250626 14:04:59.693114 3575 tablet_bootstrap.cc:654] T 7194d0f54a034d0da4f3a80eff67c28c P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:59.693153 3561 ts_tablet_manager.cc:1905] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:59.693739 3573 tablet_bootstrap.cc:654] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:59.694774 3575 tablet_bootstrap.cc:492] T 7194d0f54a034d0da4f3a80eff67c28c P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:59.694859 3575 ts_tablet_manager.cc:1397] T 7194d0f54a034d0da4f3a80eff67c28c P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 14:04:59.694895 3561 ts_tablet_manager.cc:1918] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:59.694945 3561 log.cc:1199] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-1/wal/wals/a679d473653e403584d3271e7ce68515
I20250626 14:04:59.695118 3561 ts_tablet_manager.cc:1939] T a679d473653e403584d3271e7ce68515 P e517893e13f546c0b40d976633b49651: Deleting consensus metadata
I20250626 14:04:59.695453 3575 raft_consensus.cc:357] T 7194d0f54a034d0da4f3a80eff67c28c P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:59.695588 3575 raft_consensus.cc:383] T 7194d0f54a034d0da4f3a80eff67c28c P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:59.695628 3575 raft_consensus.cc:738] T 7194d0f54a034d0da4f3a80eff67c28c P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
I20250626 14:04:59.695788 3575 consensus_queue.cc:260] T 7194d0f54a034d0da4f3a80eff67c28c P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:59.695927 2966 catalog_manager.cc:4928] TS e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): tablet a679d473653e403584d3271e7ce68515 (table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]) successfully deleted
I20250626 14:04:59.696381 3573 tablet_bootstrap.cc:492] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
@ 0x7f519ef01ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250626 14:04:59.696482 3573 ts_tablet_manager.cc:1397] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250626 14:04:59.697074 3573 raft_consensus.cc:357] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:59.697186 3573 raft_consensus.cc:383] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:59.697244 3573 raft_consensus.cc:738] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:59.697397 3573 consensus_queue.cc:260] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:59.697719 3573 ts_tablet_manager.cc:1428] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 14:04:59.698496 1994 heartbeater.cc:499] Master 127.1.11.252:37925 was elected leader, sending a full tablet report...
I20250626 14:04:59.698880 3575 ts_tablet_manager.cc:1428] T 7194d0f54a034d0da4f3a80eff67c28c P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250626 14:04:59.699108 3575 tablet_bootstrap.cc:492] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99: Bootstrap starting.
I20250626 14:04:59.699882 1859 heartbeater.cc:499] Master 127.1.11.252:37925 was elected leader, sending a full tablet report...
I20250626 14:04:59.700743 3573 tablet_bootstrap.cc:492] T 199b89801c1a4cea9ef97d73f230337b P 4e57adb83e9f451488ed1d0034043e7b: Bootstrap starting.
@ 0x7f519ef0130b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250626 14:04:59.701753 3573 tablet_bootstrap.cc:654] T 199b89801c1a4cea9ef97d73f230337b P 4e57adb83e9f451488ed1d0034043e7b: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:59.701752 3575 tablet_bootstrap.cc:654] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99: Neither blocks nor log segments found. Creating new log.
I20250626 14:04:59.702998 3575 tablet_bootstrap.cc:492] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99: No bootstrap required, opened a new log
I20250626 14:04:59.703081 3575 ts_tablet_manager.cc:1397] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250626 14:04:59.703907 3575 raft_consensus.cc:357] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:59.704027 3575 raft_consensus.cc:383] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:59.704066 3575 raft_consensus.cc:738] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Initialized, Role: FOLLOWER
@ 0x7f519ef06a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250626 14:04:59.704223 3575 consensus_queue.cc:260] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:59.704615 3575 ts_tablet_manager.cc:1428] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 14:04:59.704876 3572 tablet_replica.cc:331] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:59.704988 3572 raft_consensus.cc:2241] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Raft consensus shutting down.
@ 0x55ccfa455752 std::function<>::operator()() at ??:0
I20250626 14:04:59.705638 3573 tablet_bootstrap.cc:492] T 199b89801c1a4cea9ef97d73f230337b P 4e57adb83e9f451488ed1d0034043e7b: No bootstrap required, opened a new log
I20250626 14:04:59.705720 3573 ts_tablet_manager.cc:1397] T 199b89801c1a4cea9ef97d73f230337b P 4e57adb83e9f451488ed1d0034043e7b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250626 14:04:59.706354 3572 raft_consensus.cc:2270] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:59.706431 3573 raft_consensus.cc:357] T 199b89801c1a4cea9ef97d73f230337b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:59.706552 3573 raft_consensus.cc:383] T 199b89801c1a4cea9ef97d73f230337b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:59.706593 3573 raft_consensus.cc:738] T 199b89801c1a4cea9ef97d73f230337b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Initialized, Role: FOLLOWER
I20250626 14:04:59.706738 3573 consensus_queue.cc:260] T 199b89801c1a4cea9ef97d73f230337b P 4e57adb83e9f451488ed1d0034043e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:59.707682 3573 ts_tablet_manager.cc:1428] T 199b89801c1a4cea9ef97d73f230337b P 4e57adb83e9f451488ed1d0034043e7b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
@ 0x7f519b3bb0a1 kudu::Thread::SuperviseThread() at ??:0
I20250626 14:04:59.708566 3572 ts_tablet_manager.cc:1905] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:59.709288 3571 tablet_bootstrap.cc:492] T 7194d0f54a034d0da4f3a80eff67c28c P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:59.709385 3571 ts_tablet_manager.cc:1397] T 7194d0f54a034d0da4f3a80eff67c28c P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20250626 14:04:59.709972 3571 raft_consensus.cc:357] T 7194d0f54a034d0da4f3a80eff67c28c P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:59.710090 3571 raft_consensus.cc:383] T 7194d0f54a034d0da4f3a80eff67c28c P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:59.710130 3571 raft_consensus.cc:738] T 7194d0f54a034d0da4f3a80eff67c28c P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:59.710291 3571 consensus_queue.cc:260] T 7194d0f54a034d0da4f3a80eff67c28c P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
@ 0x7f519bbaa6db start_thread at ??:0
I20250626 14:04:59.710649 3571 ts_tablet_manager.cc:1428] T 7194d0f54a034d0da4f3a80eff67c28c P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 14:04:59.710834 3571 tablet_bootstrap.cc:492] T 199b89801c1a4cea9ef97d73f230337b P e517893e13f546c0b40d976633b49651: Bootstrap starting.
I20250626 14:04:59.712394 3571 tablet_bootstrap.cc:654] T 199b89801c1a4cea9ef97d73f230337b P e517893e13f546c0b40d976633b49651: Neither blocks nor log segments found. Creating new log.
@ 0x7f519912071f clone at ??:0
I20250626 14:04:59.714028 3571 tablet_bootstrap.cc:492] T 199b89801c1a4cea9ef97d73f230337b P e517893e13f546c0b40d976633b49651: No bootstrap required, opened a new log
I20250626 14:04:59.714100 3571 ts_tablet_manager.cc:1397] T 199b89801c1a4cea9ef97d73f230337b P e517893e13f546c0b40d976633b49651: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250626 14:04:59.714591 3571 raft_consensus.cc:357] T 199b89801c1a4cea9ef97d73f230337b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:59.714709 3571 raft_consensus.cc:383] T 199b89801c1a4cea9ef97d73f230337b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:04:59.714744 3571 raft_consensus.cc:738] T 199b89801c1a4cea9ef97d73f230337b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e517893e13f546c0b40d976633b49651, State: Initialized, Role: FOLLOWER
I20250626 14:04:59.714879 3572 ts_tablet_manager.cc:1918] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:59.714885 3571 consensus_queue.cc:260] T 199b89801c1a4cea9ef97d73f230337b P e517893e13f546c0b40d976633b49651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:59.714944 3572 log.cc:1199] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/a679d473653e403584d3271e7ce68515
I20250626 14:04:59.715202 3571 ts_tablet_manager.cc:1428] T 199b89801c1a4cea9ef97d73f230337b P e517893e13f546c0b40d976633b49651: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 14:04:59.715210 3572 ts_tablet_manager.cc:1939] T a679d473653e403584d3271e7ce68515 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:59.715843 3572 tablet_replica.cc:331] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99: stopping tablet replica
I20250626 14:04:59.715960 3572 raft_consensus.cc:2241] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 14:04:59.718812 3572 raft_consensus.cc:2270] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:04:59.719293 3572 ts_tablet_manager.cc:1905] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 14:04:59.723212 3572 ts_tablet_manager.cc:1918] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 14:04:59.723285 3572 log.cc:1199] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting WAL directory at /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/ts-2/wal/wals/0db0b418e8b8412da3dc098ef6aab940
I20250626 14:04:59.723564 3572 ts_tablet_manager.cc:1939] T 0db0b418e8b8412da3dc098ef6aab940 P 0c8941c4e30348ed9bc4a3262cd9fe99: Deleting consensus metadata
I20250626 14:04:59.835664 3437 raft_consensus.cc:491] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:59.835798 3437 raft_consensus.cc:513] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:59.836248 3437 leader_election.cc:290] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:59.836633 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "199b89801c1a4cea9ef97d73f230337b" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:59.836730 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "199b89801c1a4cea9ef97d73f230337b" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b" is_pre_election: true
I20250626 14:04:59.836779 1813 raft_consensus.cc:2466] T 199b89801c1a4cea9ef97d73f230337b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:59.836879 1678 raft_consensus.cc:2466] T 199b89801c1a4cea9ef97d73f230337b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 0.
I20250626 14:04:59.837132 1883 leader_election.cc:304] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:59.837296 3437 raft_consensus.cc:2802] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:59.837350 3437 raft_consensus.cc:491] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:59.837392 3437 raft_consensus.cc:3058] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:59.838289 3437 raft_consensus.cc:513] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:04:59.838925 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "199b89801c1a4cea9ef97d73f230337b" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:59.838925 1678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "199b89801c1a4cea9ef97d73f230337b" candidate_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e57adb83e9f451488ed1d0034043e7b"
I20250626 14:04:59.839040 1813 raft_consensus.cc:3058] T 199b89801c1a4cea9ef97d73f230337b P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:59.839047 1678 raft_consensus.cc:3058] T 199b89801c1a4cea9ef97d73f230337b P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:59.839999 1813 raft_consensus.cc:2466] T 199b89801c1a4cea9ef97d73f230337b P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:59.839999 1678 raft_consensus.cc:2466] T 199b89801c1a4cea9ef97d73f230337b P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8941c4e30348ed9bc4a3262cd9fe99 in term 1.
I20250626 14:04:59.840288 1884 leader_election.cc:304] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:59.840590 3437 leader_election.cc:290] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [CANDIDATE]: Term 1 election: Requested vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:04:59.840651 3590 raft_consensus.cc:2802] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:59.840862 3590 raft_consensus.cc:695] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8941c4e30348ed9bc4a3262cd9fe99, State: Running, Role: LEADER
I20250626 14:04:59.841046 3590 consensus_queue.cc:237] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
W20250626 14:04:59.872026 1882 connection.cc:537] server connection from 127.0.0.1:46226 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 14:04:59.873023 2001 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.11.252:37925)
W20250626 14:04:59.873186 2001 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.11.252:37925)
W20250626 14:04:59.873296 2001 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.11.252:37925)
W20250626 14:04:59.879804 2001 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.11.252:37925)
I20250626 14:04:59.927557 3359 raft_consensus.cc:491] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:04:59.927707 3359 raft_consensus.cc:513] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:59.928167 3359 leader_election.cc:290] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:59.928694 1948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7194d0f54a034d0da4f3a80eff67c28c" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" is_pre_election: true
I20250626 14:04:59.928745 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7194d0f54a034d0da4f3a80eff67c28c" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651" is_pre_election: true
I20250626 14:04:59.928850 1948 raft_consensus.cc:2466] T 7194d0f54a034d0da4f3a80eff67c28c P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 0.
I20250626 14:04:59.928908 1813 raft_consensus.cc:2466] T 7194d0f54a034d0da4f3a80eff67c28c P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 0.
I20250626 14:04:59.929334 1613 leader_election.cc:304] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b, e517893e13f546c0b40d976633b49651; no voters:
I20250626 14:04:59.929563 3359 raft_consensus.cc:2802] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:04:59.929643 3359 raft_consensus.cc:491] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:04:59.929697 3359 raft_consensus.cc:3058] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:59.930557 3359 raft_consensus.cc:513] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:04:59.930862 3359 leader_election.cc:290] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 election: Requested vote from peers 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231), e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
I20250626 14:04:59.931211 1948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7194d0f54a034d0da4f3a80eff67c28c" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99"
I20250626 14:04:59.931334 1948 raft_consensus.cc:3058] T 7194d0f54a034d0da4f3a80eff67c28c P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:59.931315 1813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7194d0f54a034d0da4f3a80eff67c28c" candidate_uuid: "4e57adb83e9f451488ed1d0034043e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e517893e13f546c0b40d976633b49651"
I20250626 14:04:59.931438 1813 raft_consensus.cc:3058] T 7194d0f54a034d0da4f3a80eff67c28c P e517893e13f546c0b40d976633b49651 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:04:59.932149 1948 raft_consensus.cc:2466] T 7194d0f54a034d0da4f3a80eff67c28c P 0c8941c4e30348ed9bc4a3262cd9fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
I20250626 14:04:59.932233 1813 raft_consensus.cc:2466] T 7194d0f54a034d0da4f3a80eff67c28c P e517893e13f546c0b40d976633b49651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e57adb83e9f451488ed1d0034043e7b in term 1.
I20250626 14:04:59.932464 1613 leader_election.cc:304] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8941c4e30348ed9bc4a3262cd9fe99, 4e57adb83e9f451488ed1d0034043e7b; no voters:
I20250626 14:04:59.932643 3359 raft_consensus.cc:2802] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:04:59.932732 3359 raft_consensus.cc:695] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [term 1 LEADER]: Becoming Leader. State: Replica: 4e57adb83e9f451488ed1d0034043e7b, State: Running, Role: LEADER
I20250626 14:04:59.932888 3359 consensus_queue.cc:237] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } } peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } }
I20250626 14:05:00.181434 3596 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 81db61eed5584e7e8b0cf40e5441f88e)
I20250626 14:05:00.181607 3596 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:00.182863 3596 leader_election.cc:290] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:00.183631 3469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
W20250626 14:05:00.184499 3469 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
W20250626 14:05:00.184671 3469 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:05:00.184715 3469 leader_election.cc:304] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19; no voters: 24ad911d71b6433f8c7939a1d1f7b782, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:00.184871 3596 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250626 14:05:00.284585 3591 consensus_queue.cc:1035] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:05:00.288314 3590 consensus_queue.cc:1035] T 199b89801c1a4cea9ef97d73f230337b P 0c8941c4e30348ed9bc4a3262cd9fe99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:05:00.371104 3360 consensus_queue.cc:1035] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:05:00.375905 3360 consensus_queue.cc:1035] T 7194d0f54a034d0da4f3a80eff67c28c P 4e57adb83e9f451488ed1d0034043e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:05:00.717105 3360 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:05:00.717267 3360 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:05:00.717746 3360 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:05:00.718528 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:05:00.718643 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:05:00.718680 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:05:00.718823 3360 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 14:05:00.790390 3609 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 81db61eed5584e7e8b0cf40e5441f88e)
I20250626 14:05:00.790527 3609 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:00.790953 3609 leader_election.cc:290] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:00.792343 3469 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
W20250626 14:05:00.792496 3469 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:05:00.792575 3469 leader_election.cc:304] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19; no voters: 24ad911d71b6433f8c7939a1d1f7b782, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:00.792728 3609 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250626 14:05:00.952786 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 26: this message will repeat every 5th retry.
I20250626 14:05:01.376708 3619 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 81db61eed5584e7e8b0cf40e5441f88e)
I20250626 14:05:01.376834 3619 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:01.377247 3619 leader_election.cc:290] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:01.378597 3469 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707): Network error: Client connection negotiation failed: client connection to 127.1.11.254:36707: connect: Connection refused (error 111)
W20250626 14:05:01.378726 3469 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:05:01.378775 3469 leader_election.cc:304] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19; no voters: 24ad911d71b6433f8c7939a1d1f7b782, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:01.378909 3619 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250626 14:05:01.821872 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 26: this message will repeat every 5th retry.
W20250626 14:05:01.984779 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 26: this message will repeat every 5th retry.
I20250626 14:05:02.292184 1071 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmpJjah/build/debug/bin/kudu
/tmp/dist-test-taskmpJjah/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.254:36707
--webserver_interface=127.1.11.254
--webserver_port=45241
--builtin_ntp_servers=127.1.11.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250626 14:05:02.402482 3628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:05:02.402710 3628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:05:02.402750 3628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:05:02.406049 3628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:05:02.406116 3628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:05:02.406150 3628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:05:02.406193 3628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:05:02.410073 3628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.11.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:41657
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.11.254:36707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.11.254
--webserver_port=45241
--never_fsync=true
--heap_profile_path=/tmp/kudu.3628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type DEBUG
built by None at 26 Jun 2025 13:43:14 UTC on 24a791456cd2
build id 6778
I20250626 14:05:02.411204 3628 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:05:02.412346 3628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:05:02.420190 3638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:05:02.420303 3635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:05:02.420190 3636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:05:02.420544 3628 server_base.cc:1048] running on GCE node
I20250626 14:05:02.421240 3628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:05:02.422142 3628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:05:02.423309 3628 hybrid_clock.cc:648] HybridClock initialized: now 1750946702423302 us; error 25 us; skew 500 ppm
I20250626 14:05:02.430980 3628 webserver.cc:469] Webserver started at http://127.1.11.254:45241/ using document root <none> and password file <none>
I20250626 14:05:02.431602 3628 fs_manager.cc:362] Metadata directory not provided
I20250626 14:05:02.431668 3628 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:05:02.435163 3628 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250626 14:05:02.437278 3644 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:05:02.438324 3628 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250626 14:05:02.438432 3628 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data,/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
uuid: "24ad911d71b6433f8c7939a1d1f7b782"
format_stamp: "Formatted at 2025-06-26 14:04:30 on dist-test-slave-rptj"
I20250626 14:05:02.438831 3628 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:05:02.452006 3628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:05:02.452716 3628 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:05:02.452884 3628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:05:02.460193 3628 rpc_server.cc:307] RPC server started. Bound to: 127.1.11.254:36707
I20250626 14:05:02.460269 3696 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.11.254:36707 every 8 connection(s)
I20250626 14:05:02.461377 3628 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/master-0/data/info.pb
I20250626 14:05:02.465409 3697 sys_catalog.cc:263] Verifying existing consensus state
I20250626 14:05:02.467067 3697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Bootstrap starting.
I20250626 14:05:02.468400 1071 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmpJjah/build/debug/bin/kudu as pid 3628
I20250626 14:05:02.478531 3697 log.cc:826] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Log is configured to *not* fsync() on all Append() calls
I20250626 14:05:02.504489 1860 heartbeater.cc:344] Connected to a master server at 127.1.11.254:36707
I20250626 14:05:02.521766 3697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Bootstrap replayed 1/1 log segments. Stats: ops{read=72 overwritten=0 applied=71 ignored=0} inserts{seen=49 ignored=0} mutations{seen=58 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250626 14:05:02.522395 3697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: Bootstrap complete.
I20250626 14:05:02.525146 3697 raft_consensus.cc:357] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 5 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:02.525683 3697 raft_consensus.cc:738] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ad911d71b6433f8c7939a1d1f7b782, State: Initialized, Role: FOLLOWER
I20250626 14:05:02.526154 3697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 5.72, Last appended by leader: 72, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:02.526610 3701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:05:02.526733 3701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 14:05:02.527263 3697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:05:02.534144 3715 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:05:02.534231 3715 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: loading cluster ID for follower catalog manager: success
I20250626 14:05:02.537006 3715 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: acquiring CA information for follower catalog manager: success
I20250626 14:05:02.538455 3715 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 14:05:02.548305 3628 master_runner.cc:186] Error getting master registration for 127.1.11.252:37925: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111),
I20250626 14:05:02.550446 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:05:02.822686 3661 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e517893e13f546c0b40d976633b49651" instance_seqno: 1750946671821196) as {username='slave'} at 127.1.11.194:46949; Asking this server to re-register.
I20250626 14:05:02.823283 1860 heartbeater.cc:461] Registering TS with master...
I20250626 14:05:02.824471 3661 ts_manager.cc:194] Registered new tserver with Master: e517893e13f546c0b40d976633b49651 (127.1.11.194:44397)
W20250626 14:05:02.845974 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 26: this message will repeat every 5th retry.
I20250626 14:05:02.849046 1725 heartbeater.cc:344] Connected to a master server at 127.1.11.254:36707
I20250626 14:05:02.850252 3661 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" instance_seqno: 1750946671637378) as {username='slave'} at 127.1.11.193:45279; Asking this server to re-register.
I20250626 14:05:02.850663 1725 heartbeater.cc:461] Registering TS with master...
I20250626 14:05:02.851593 3661 ts_manager.cc:194] Registered new tserver with Master: 4e57adb83e9f451488ed1d0034043e7b (127.1.11.193:35671)
I20250626 14:05:02.869151 3360 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:05:02.869320 3360 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:05:02.869778 3360 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:05:02.870548 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:05:02.870855 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:05:02.870931 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:05:02.871095 3360 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 14:05:02.921960 1995 heartbeater.cc:344] Connected to a master server at 127.1.11.254:36707
I20250626 14:05:02.923233 3661 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" instance_seqno: 1750946672002466) as {username='slave'} at 127.1.11.195:48303; Asking this server to re-register.
I20250626 14:05:02.923754 1995 heartbeater.cc:461] Registering TS with master...
I20250626 14:05:02.924732 3661 ts_manager.cc:194] Registered new tserver with Master: 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
I20250626 14:05:02.956315 3721 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 81db61eed5584e7e8b0cf40e5441f88e)
I20250626 14:05:02.956460 3721 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:02.956910 3721 leader_election.cc:290] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:02.958410 3469 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:05:02.960717 3671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07532b7648e4414789ed2b099bad2b19" candidate_term: 9 candidate_status { last_received { term: 8 index: 89 } } ignore_live_leader: false dest_uuid: "24ad911d71b6433f8c7939a1d1f7b782" is_pre_election: true
I20250626 14:05:02.961014 3671 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07532b7648e4414789ed2b099bad2b19 in term 5.
I20250626 14:05:02.961572 3469 leader_election.cc:304] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782; no voters: 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:02.961755 3721 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250626 14:05:02.961815 3721 raft_consensus.cc:491] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 81db61eed5584e7e8b0cf40e5441f88e)
I20250626 14:05:02.961872 3721 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 8 FOLLOWER]: Advancing to term 9
I20250626 14:05:02.962929 3721 raft_consensus.cc:513] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:02.963311 3721 leader_election.cc:290] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 election: Requested vote from peers 24ad911d71b6433f8c7939a1d1f7b782 (127.1.11.254:36707), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
I20250626 14:05:02.963629 3671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07532b7648e4414789ed2b099bad2b19" candidate_term: 9 candidate_status { last_received { term: 8 index: 89 } } ignore_live_leader: false dest_uuid: "24ad911d71b6433f8c7939a1d1f7b782"
I20250626 14:05:02.963760 3671 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 5 FOLLOWER]: Advancing to term 9
W20250626 14:05:02.964424 3469 leader_election.cc:336] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:05:02.965781 3671 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07532b7648e4414789ed2b099bad2b19 in term 9.
I20250626 14:05:02.966086 3469 leader_election.cc:304] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07532b7648e4414789ed2b099bad2b19, 24ad911d71b6433f8c7939a1d1f7b782; no voters: 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:02.966266 3721 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 9 FOLLOWER]: Leader election won for term 9
I20250626 14:05:02.966482 3721 raft_consensus.cc:695] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [term 9 LEADER]: Becoming Leader. State: Replica: 07532b7648e4414789ed2b099bad2b19, State: Running, Role: LEADER
I20250626 14:05:02.966765 3721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 8.89, Last appended by leader: 89, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:02.968218 3725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07532b7648e4414789ed2b099bad2b19. Latest consensus state: current_term: 9 leader_uuid: "07532b7648e4414789ed2b099bad2b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } } }
I20250626 14:05:02.968336 3725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19 [sys.catalog]: This master's current role is: LEADER
I20250626 14:05:02.968683 3727 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 14:05:02.970379 3727 catalog_manager.cc:671] Loaded metadata for table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03]
I20250626 14:05:02.970664 3727 catalog_manager.cc:671] Loaded metadata for table default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843]
I20250626 14:05:02.970842 3727 catalog_manager.cc:671] Loaded metadata for table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a]
I20250626 14:05:02.971094 3727 catalog_manager.cc:671] Loaded metadata for table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb]
I20250626 14:05:02.971252 3727 catalog_manager.cc:671] Loaded metadata for table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9]
I20250626 14:05:02.971406 3727 catalog_manager.cc:671] Loaded metadata for table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80]
I20250626 14:05:02.971570 3727 catalog_manager.cc:671] Loaded metadata for table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75]
I20250626 14:05:02.971728 3727 catalog_manager.cc:671] Loaded metadata for table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233]
I20250626 14:05:02.971879 3727 catalog_manager.cc:671] Loaded metadata for table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447]
I20250626 14:05:02.972047 3727 catalog_manager.cc:671] Loaded metadata for table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c]
I20250626 14:05:02.972290 3727 catalog_manager.cc:671] Loaded metadata for table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140]
I20250626 14:05:02.973618 3727 tablet_loader.cc:96] loaded metadata for tablet 025415f7a05746e19d5ed199246fc949 (table default.table_f9b5b5e2dcf64f9e889c618242274d25 [id=457542e389754e82bfcab4452db8557a])
I20250626 14:05:02.974354 3727 tablet_loader.cc:96] loaded metadata for tablet 199b89801c1a4cea9ef97d73f230337b (table default.table_22f5000aea364833a97948c4478f7e27 [id=8bc8b25d9539413ea196c158d55abe80])
I20250626 14:05:02.974659 3727 tablet_loader.cc:96] loaded metadata for tablet 33d002d179974a95aaad3d84814d27f0 (table default.table_fb8ebc7aa8894f28b4d79252382f356a [id=9177a82a73394ff798d7735ed15f9e75])
I20250626 14:05:02.974951 3727 tablet_loader.cc:96] loaded metadata for tablet 48adfabcc2d84b6a9bfb7134f2c860ce (table default.table_42d07c2b16314b1999eea5f3878944fe [id=61127b0a117a4fbaa8bbbbb397cbc6bb])
I20250626 14:05:02.975653 3727 tablet_loader.cc:96] loaded metadata for tablet 7194d0f54a034d0da4f3a80eff67c28c (table default.table_3a3f21b3c0e54e438561e02ab79fa36b [id=fb8ec39fc8a34deca5b53c18cc207140])
I20250626 14:05:02.975793 3727 tablet_loader.cc:96] loaded metadata for tablet 730ed7238006463eb08803a70330a43f (table default.table_72f9307d2f5247a0ae4499c953ea1650 [id=61be84e63b2e478b9da5619fdbdffee9])
I20250626 14:05:02.975992 3727 tablet_loader.cc:96] loaded metadata for tablet 7d559ff794ca4485bc682b6b8f5b8e0e (table default.table_afd26acd01c7422fbb06520af5a4f76c [id=aa8a6a6f408d454abdc5948e21f84233])
I20250626 14:05:02.976213 3727 tablet_loader.cc:96] loaded metadata for tablet 94c7591e52cd4d88946f7e98f15e2b94 (table default.table_e6e7d4c291af4d29baf43e204c5292f0 [id=c3aa594eee4244029a38801f34b07447])
I20250626 14:05:02.976708 3727 tablet_loader.cc:96] loaded metadata for tablet b25221169c9c48718276736fe657f378 (table default.table_f98cc740eae3444b88bb688eb8c8a316 [id=2d38a93c3aca41cfb07d20ca6241d843])
I20250626 14:05:02.976925 3727 tablet_loader.cc:96] loaded metadata for tablet b85960777e004b88850a7c16df33a4fd (table default.table_0be8dc44c93040319885961d6e4a0608 [id=2339533510b347b9bed82553187bab03])
I20250626 14:05:02.977051 3727 tablet_loader.cc:96] loaded metadata for tablet c744a0827e7943ba9a6ac09cf5e0a9bd (table default.table_0b500f44e16e45e589c01ab7ad9a621b [id=cdd0994532564911854d5ce33557b62c])
I20250626 14:05:02.977789 3727 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 14:05:02.978551 3727 catalog_manager.cc:1261] Loaded cluster ID: 28727086768143b6b0c9b5a907b8e86f
I20250626 14:05:02.978600 3727 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 14:05:02.979660 3727 catalog_manager.cc:1506] Loading token signing keys...
I20250626 14:05:02.980518 3727 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 07532b7648e4414789ed2b099bad2b19: Loaded TSK: 0
I20250626 14:05:02.981292 3727 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 14:05:02.981894 3727 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250626 14:05:02.982436 3727 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 13
I20250626 14:05:03.060750 3484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_afd26acd01c7422fbb06520af5a4f76c" } new_table_name: "default.Table_4d81e86bd6614c3db77efeaca0daa56f" modify_external_catalogs: true
I20250626 14:05:03.060928 3480 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42302:
name: "default.Table_5bf40f09daaa4240a8bcab30e1aa341f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:05:03.061105 3483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_42d07c2b16314b1999eea5f3878944fe" } modify_external_catalogs: true
I20250626 14:05:03.061206 3483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_42d07c2b16314b1999eea5f3878944fe" } modify_external_catalogs: true
I20250626 14:05:03.061403 3482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42302:
name: "default.Table_f7012cd4bf5d40d6af93cf9b01405dcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:05:03.061578 3476 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42302:
name: "default.Table_33ee3521f64d4cefb8856259510e5f0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:05:03.061662 3481 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_3a3f21b3c0e54e438561e02ab79fa36b" } new_table_name: "default.Table_3633fc38c5374180a081fab63a1eed84" modify_external_catalogs: true
I20250626 14:05:03.061844 3477 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_0b500f44e16e45e589c01ab7ad9a621b" } modify_external_catalogs: true
I20250626 14:05:03.061884 3479 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_f9b5b5e2dcf64f9e889c618242274d25" } modify_external_catalogs: true
I20250626 14:05:03.061944 3477 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_0b500f44e16e45e589c01ab7ad9a621b" } modify_external_catalogs: true
I20250626 14:05:03.062062 3478 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_e6e7d4c291af4d29baf43e204c5292f0" } new_table_name: "default.Table_f01c05221e5c4e75bf700b9474072881" modify_external_catalogs: true
I20250626 14:05:03.060767 3485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42302:
name: "default.Table_34cb28e2c8d740aba5c52377533fdfa4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:05:03.061951 3479 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_f9b5b5e2dcf64f9e889c618242274d25" } modify_external_catalogs: true
F20250626 14:05:03.065424 3541 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 16, previous 16; dumping first 4 out of 8 received events: NotificationEvent(eventId=14, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_72f9307d2f5247a0ae4499c953ea1650, message={"server":"","servicePrincipal":"","db":"default","table":"table_72f9307d2f5247a0ae4499c953ea1650","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_72f9307d2f5247a0ae4499c953ea1650\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_72f9307d2f5247a0ae4499c953ea1650\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_72f9307d2f5247a0ae4499c953ea1650\",\"kudu.table_id\":\"61be84e63b2e478b9da5619fdbdffee9\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=15, eventTime=1750946674, eventType=CREATE_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=ALTER_TABLE, dbName=default, tableName=table_7d5a9b0c3ae440b89dac3ca5dd8e81ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_fb8ebc7aa8894f28b4d79252382f356a","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_fb8ebc7aa8894f28b4d79252382f356a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_fb8ebc7aa8894f28b4d79252382f356a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fb8ebc7aa8894f28b4d79252382f356a\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946672},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7d5a9b0c3ae440b89dac3ca5dd8e81ef\",\"kudu.table_id\":\"9177a82a73394ff798d7735ed15f9e75\",\"transient_lastDdlTime\":\"1750946672\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750946674}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=16, eventTime=1750946674, eventType=DROP_TABLE, dbName=default, tableName=table_2c8dee0163784a60a8468b53619247f4, message={"server":"","servicePrincipal":"","db":"default","table":"table_2c8dee0163784a60a8468b53619247f4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_2c8dee0163784a60a8468b53619247f4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750946674},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskmpJjah/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750946656173073-1071-0/minicluster-data/warehouse/table_2c8dee0163784a60a8468b53619247f4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_2c8dee0163784a60a8468b53619247f4\",\"kudu.table_id\":\"6784347b414a45f785c279bf0ab4800e\",\"transient_lastDdlTime\":\"1750946674\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"28727086768143b6b0c9b5a907b8e86f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750946674}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f30bd7d4dcd google::LogMessage::Fail() at ??:0
@ 0x7f30bd7d8b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f30bd7d47cc google::LogMessage::Flush() at ??:0
@ 0x7f30bd7d4a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f30c276dc91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f30c276cba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f30c276c30b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7f30c2771a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55d34ece0752 std::function<>::operator()() at ??:0
@ 0x7f30bec260a1 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f30bf4156db start_thread at ??:0
@ 0x7f30bc98b71f clone at ??:0
W20250626 14:05:03.246383 2003 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.11.253:44447)
W20250626 14:05:03.246776 2003 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.11.253:44447)
W20250626 14:05:03.246878 2003 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.11.253:44447)
W20250626 14:05:03.246966 2003 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.11.253:44447)
I20250626 14:05:03.678537 3744 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:05:03.678678 3744 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:03.679874 3744 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:03.680476 3647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:05:03.681190 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:05:03.681298 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:05:03.681365 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:03.681548 3744 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250626 14:05:04.251760 3748 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:05:04.251884 3748 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:04.252386 3748 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:04.253674 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:05:04.253773 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:05:04.253863 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:04.254002 3748 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250626 14:05:04.900128 1071 master-stress-test.cc:577] Killing test thread 1/25
I20250626 14:05:04.923640 3759 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:05:04.923769 3759 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:04.924214 3759 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:04.925987 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:05:04.926112 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:05:04.926196 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:04.926342 3759 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:05:05.840917 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 31: this message will repeat every 5th retry.
I20250626 14:05:05.941444 3359 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:05:05.941581 3359 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:05:05.941928 3359 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:05:05.942610 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:05:05.942764 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:05:05.942821 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:05:05.942986 3359 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:05:06.462774 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 31: this message will repeat every 5th retry.
I20250626 14:05:06.540997 3773 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:05:06.541141 3773 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:06.541566 3773 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:06.542694 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:05:06.542783 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:05:06.542868 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:06.542996 3773 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:05:06.784726 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 31: this message will repeat every 5th retry.
I20250626 14:05:07.557535 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:05:07.557646 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:05:07.947638 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 31: this message will repeat every 5th retry.
I20250626 14:05:09.039917 3793 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:05:09.040063 3793 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:09.040505 3793 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:09.041126 3647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20250626 14:05:09.041908 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:05:09.041994 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:05:09.042076 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:09.042213 3793 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250626 14:05:10.432058 3360 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:05:10.432215 3360 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:05:10.432833 3360 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:05:10.433954 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:05:10.434085 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:05:10.434140 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:05:10.434286 3360 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:05:11.266692 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 36: this message will repeat every 5th retry.
W20250626 14:05:11.287184 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 36: this message will repeat every 5th retry.
I20250626 14:05:11.798593 3811 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:05:11.798731 3811 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:11.799216 3811 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:11.800398 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:05:11.800498 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:05:11.800613 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:11.800766 3811 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:05:12.315848 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 36: this message will repeat every 5th retry.
I20250626 14:05:12.559525 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:05:12.559648 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:05:12.695618 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 36: this message will repeat every 5th retry.
I20250626 14:05:16.004317 3846 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:05:16.004515 3846 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:16.004995 3846 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:16.005589 3645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250626 14:05:16.006269 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:05:16.006410 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:05:16.006470 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:16.006613 3846 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:05:16.361533 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 41: this message will repeat every 5th retry.
W20250626 14:05:16.807813 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 41: this message will repeat every 5th retry.
W20250626 14:05:17.198343 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 41: this message will repeat every 5th retry.
I20250626 14:05:17.340291 3828 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:05:17.340425 3828 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:05:17.340863 3828 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:05:17.341523 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:05:17.341647 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:05:17.341701 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:05:17.341863 3828 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 14:05:17.561640 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:05:17.561771 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:05:18.128962 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 41: this message will repeat every 5th retry.
I20250626 14:05:19.501153 3872 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:05:19.501296 3872 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:19.501780 3872 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:19.502971 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:05:19.503070 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:05:19.503156 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:19.503293 3872 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:05:21.142230 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 46: this message will repeat every 5th retry.
W20250626 14:05:21.680469 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 46: this message will repeat every 5th retry.
I20250626 14:05:22.563917 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:05:22.564039 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:05:22.590822 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 46: this message will repeat every 5th retry.
W20250626 14:05:22.993535 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 46: this message will repeat every 5th retry.
I20250626 14:05:24.344897 3909 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:05:24.345039 3909 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:24.345468 3909 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:24.346060 3647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250626 14:05:24.346827 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:05:24.347000 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:05:24.347074 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:24.347234 3909 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250626 14:05:25.505045 3899 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:05:25.505188 3899 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:05:25.505729 3899 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:05:25.506415 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:05:25.506537 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:05:25.506583 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:05:25.506726 3899 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:05:26.570811 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 51: this message will repeat every 5th retry.
W20250626 14:05:26.724256 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 51: this message will repeat every 5th retry.
W20250626 14:05:27.495366 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 51: this message will repeat every 5th retry.
I20250626 14:05:27.566118 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:05:27.566236 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:05:28.448750 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 51: this message will repeat every 5th retry.
W20250626 14:05:31.316633 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 56: this message will repeat every 5th retry.
W20250626 14:05:32.111047 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 56: this message will repeat every 5th retry.
W20250626 14:05:32.182686 1724 heartbeater.cc:646] Failed to heartbeat to 127.1.11.252:37925 (139 consecutive failures): Network error: Failed to ping master at 127.1.11.252:37925: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:05:32.187481 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 56: this message will repeat every 5th retry.
W20250626 14:05:32.566905 1861 heartbeater.cc:646] Failed to heartbeat to 127.1.11.253:44447 (116 consecutive failures): Network error: Failed to ping master at 127.1.11.253:44447: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:05:32.568115 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:05:32.568215 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:05:32.977902 1996 heartbeater.cc:646] Failed to heartbeat to 127.1.11.253:44447 (59 consecutive failures): Network error: Failed to ping master at 127.1.11.253:44447: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:05:33.346237 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 56: this message will repeat every 5th retry.
W20250626 14:05:36.497382 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 61: this message will repeat every 5th retry.
W20250626 14:05:37.124614 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 61: this message will repeat every 5th retry.
W20250626 14:05:37.152957 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 61: this message will repeat every 5th retry.
I20250626 14:05:37.570132 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:05:37.570242 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:05:38.315166 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 61: this message will repeat every 5th retry.
I20250626 14:05:38.532104 3997 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:05:38.532238 3997 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:05:38.532646 3997 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:05:38.533206 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:05:38.533346 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:05:38.533394 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:05:38.533533 3997 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 14:05:39.807121 4019 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:05:39.807276 4019 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:39.807804 4019 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:39.808452 3647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250626 14:05:39.808452 3645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:05:39.809299 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:05:39.809448 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:05:39.809511 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:39.809682 4019 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:05:41.569415 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 66: this message will repeat every 5th retry.
W20250626 14:05:41.963812 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 66: this message will repeat every 5th retry.
W20250626 14:05:42.450186 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 66: this message will repeat every 5th retry.
I20250626 14:05:42.572088 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:05:42.572204 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:05:43.609194 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 66: this message will repeat every 5th retry.
I20250626 14:05:45.722491 3898 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:05:45.722630 3898 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:05:45.723145 3898 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:05:45.723920 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:05:45.724067 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:05:45.724116 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:05:45.724325 3898 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:05:46.581110 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 71: this message will repeat every 5th retry.
W20250626 14:05:46.996469 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 71: this message will repeat every 5th retry.
W20250626 14:05:47.530534 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 71: this message will repeat every 5th retry.
I20250626 14:05:47.574390 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:05:47.574545 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:05:48.892938 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 71: this message will repeat every 5th retry.
W20250626 14:05:51.551568 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 76: this message will repeat every 5th retry.
W20250626 14:05:51.729928 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 76: this message will repeat every 5th retry.
W20250626 14:05:52.458211 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 76: this message will repeat every 5th retry.
I20250626 14:05:52.576561 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:05:52.576679 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:05:53.768606 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 76: this message will repeat every 5th retry.
W20250626 14:05:56.441078 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 81: this message will repeat every 5th retry.
W20250626 14:05:56.679925 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 81: this message will repeat every 5th retry.
W20250626 14:05:57.182638 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 81: this message will repeat every 5th retry.
I20250626 14:05:57.578751 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:05:57.578886 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:05:57.602152 4145 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:05:57.602291 4145 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:57.602888 4145 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:57.603276 3647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:05:57.604094 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:05:57.604220 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:05:57.604285 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:57.604432 4145 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250626 14:05:58.730955 4157 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:05:58.731081 4157 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:05:58.731546 4157 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:05:58.732625 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:05:58.732712 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:05:58.732795 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:05:58.732941 4157 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:05:58.736002 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 81: this message will repeat every 5th retry.
I20250626 14:05:59.676086 4119 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:05:59.676213 4119 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:05:59.676643 4119 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:05:59.677426 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:05:59.677551 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:05:59.677624 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:05:59.677772 4119 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:06:01.352406 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 86: this message will repeat every 5th retry.
W20250626 14:06:01.616338 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 86: this message will repeat every 5th retry.
W20250626 14:06:01.787268 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 86: this message will repeat every 5th retry.
I20250626 14:06:02.097709 4118 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:06:02.097853 4118 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:06:02.098330 4118 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:06:02.099047 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:06:02.099161 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:06:02.099203 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:06:02.099371 4118 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 14:06:02.170859 4181 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:06:02.171057 4181 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:06:02.171448 4181 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:06:02.172724 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:06:02.172963 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:06:02.173033 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:06:02.173167 4181 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250626 14:06:02.580715 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:06:02.580822 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:06:03.832656 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 86: this message will repeat every 5th retry.
W20250626 14:06:06.444715 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 91: this message will repeat every 5th retry.
W20250626 14:06:06.504561 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 91: this message will repeat every 5th retry.
W20250626 14:06:06.908444 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 91: this message will repeat every 5th retry.
I20250626 14:06:07.582930 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:06:07.583027 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:06:09.046136 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 91: this message will repeat every 5th retry.
W20250626 14:06:11.276060 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 96: this message will repeat every 5th retry.
W20250626 14:06:11.650781 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 96: this message will repeat every 5th retry.
W20250626 14:06:11.878989 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 96: this message will repeat every 5th retry.
I20250626 14:06:12.585266 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:06:12.585388 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:06:13.380168 4221 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:06:13.380337 4221 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:06:13.380843 4221 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:06:13.381784 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:06:13.381927 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:06:13.381979 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:06:13.382181 4221 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:06:13.528337 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 96: this message will repeat every 5th retry.
W20250626 14:06:16.064695 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 101: this message will repeat every 5th retry.
W20250626 14:06:16.565685 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 101: this message will repeat every 5th retry.
W20250626 14:06:16.900617 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 101: this message will repeat every 5th retry.
I20250626 14:06:17.587529 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:06:17.587641 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:06:18.652098 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 101: this message will repeat every 5th retry.
I20250626 14:06:19.320921 4325 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:06:19.321050 4325 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:06:19.321542 4325 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:06:19.322176 3647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250626 14:06:19.322890 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:06:19.322968 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:06:19.323027 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:06:19.323150 4325 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:06:20.842664 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 106: this message will repeat every 5th retry.
W20250626 14:06:21.597090 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 106: this message will repeat every 5th retry.
I20250626 14:06:22.166685 4357 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:06:22.166841 4357 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:06:22.167312 4357 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:06:22.168478 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:06:22.168588 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:06:22.168674 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:06:22.168821 4357 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:06:22.310232 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 106: this message will repeat every 5th retry.
I20250626 14:06:22.589533 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:06:22.589645 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:06:23.723469 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 106: this message will repeat every 5th retry.
W20250626 14:06:25.636711 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 111: this message will repeat every 5th retry.
I20250626 14:06:26.265451 4363 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:06:26.265614 4363 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:06:26.266116 4363 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:06:26.266961 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:06:26.267082 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:06:26.267117 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:06:26.267280 4363 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:06:26.387804 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 111: this message will repeat every 5th retry.
W20250626 14:06:27.370247 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 111: this message will repeat every 5th retry.
I20250626 14:06:27.591358 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:06:27.591472 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:06:28.404306 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 111: this message will repeat every 5th retry.
W20250626 14:06:30.824887 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 116: this message will repeat every 5th retry.
W20250626 14:06:31.170660 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 116: this message will repeat every 5th retry.
W20250626 14:06:32.286176 1724 heartbeater.cc:646] Failed to heartbeat to 127.1.11.252:37925 (389 consecutive failures): Network error: Failed to ping master at 127.1.11.252:37925: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:06:32.286231 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 116: this message will repeat every 5th retry.
I20250626 14:06:32.593685 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:06:32.593811 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:06:32.834318 1859 heartbeater.cc:646] Failed to heartbeat to 127.1.11.252:37925 (381 consecutive failures): Network error: Failed to ping master at 127.1.11.252:37925: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:06:33.097954 1996 heartbeater.cc:646] Failed to heartbeat to 127.1.11.253:44447 (179 consecutive failures): Network error: Failed to ping master at 127.1.11.253:44447: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:06:33.606251 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 116: this message will repeat every 5th retry.
W20250626 14:06:36.036229 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 121: this message will repeat every 5th retry.
W20250626 14:06:36.274586 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 121: this message will repeat every 5th retry.
I20250626 14:06:37.595629 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:06:37.595731 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:06:37.719451 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 121: this message will repeat every 5th retry.
W20250626 14:06:38.140910 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 121: this message will repeat every 5th retry.
I20250626 14:06:40.079156 4478 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:06:40.079322 4478 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:06:40.079911 4478 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:06:40.080509 3645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250626 14:06:40.081202 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:06:40.081298 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:06:40.081391 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:06:40.081554 4478 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:06:41.055370 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 126: this message will repeat every 5th retry.
W20250626 14:06:41.142750 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 126: this message will repeat every 5th retry.
I20250626 14:06:42.598016 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:06:42.598176 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:06:42.956795 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 126: this message will repeat every 5th retry.
W20250626 14:06:43.133020 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 126: this message will repeat every 5th retry.
I20250626 14:06:43.879504 4515 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:06:43.879659 4515 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:06:43.880077 4515 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:06:43.881356 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:06:43.881459 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:06:43.881537 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:06:43.881692 4515 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250626 14:06:44.656454 4412 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:06:44.656656 4412 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:06:44.657223 4412 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:06:44.658070 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:06:44.658191 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:06:44.658247 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:06:44.658429 4412 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:06:46.077353 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 131: this message will repeat every 5th retry.
W20250626 14:06:46.188436 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 131: this message will repeat every 5th retry.
I20250626 14:06:46.775897 4363 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:06:46.776059 4363 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:06:46.776512 4363 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:06:46.777156 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:06:46.777273 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:06:46.777326 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:06:46.777493 4363 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 14:06:47.600090 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:06:47.600205 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:06:47.889323 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 131: this message will repeat every 5th retry.
W20250626 14:06:47.937793 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 131: this message will repeat every 5th retry.
W20250626 14:06:50.870380 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 136: this message will repeat every 5th retry.
W20250626 14:06:51.082273 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 136: this message will repeat every 5th retry.
I20250626 14:06:51.214429 4569 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:06:51.214566 4569 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:06:51.214960 4569 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:06:51.215623 3645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250626 14:06:51.216348 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:06:51.216439 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:06:51.216557 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:06:51.216747 4569 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250626 14:06:52.602051 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:06:52.602191 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:06:52.817701 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 136: this message will repeat every 5th retry.
W20250626 14:06:52.903440 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 136: this message will repeat every 5th retry.
W20250626 14:06:55.801641 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 141: this message will repeat every 5th retry.
W20250626 14:06:56.329835 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 141: this message will repeat every 5th retry.
I20250626 14:06:57.604111 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:06:57.604234 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:06:57.709422 4565 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:06:57.709558 4565 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:06:57.709954 4565 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:06:57.710680 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:06:57.710858 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:06:57.710917 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:06:57.711062 4565 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:06:57.875142 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 141: this message will repeat every 5th retry.
W20250626 14:06:57.948505 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 141: this message will repeat every 5th retry.
W20250626 14:07:00.932570 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 146: this message will repeat every 5th retry.
W20250626 14:07:01.269622 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 146: this message will repeat every 5th retry.
I20250626 14:07:02.606446 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:07:02.606557 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:07:02.614512 3650 maintenance_manager.cc:419] P 24ad911d71b6433f8c7939a1d1f7b782: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033358
I20250626 14:07:02.657563 3649 maintenance_manager.cc:643] P 24ad911d71b6433f8c7939a1d1f7b782: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.043s user 0.012s sys 0.006s Metrics: {"bytes_written":18115,"cfile_init":1,"dirs.queue_time_us":237,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1091,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":440,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":45,"thread_start_us":170,"threads_started":2,"wal-append.queue_time_us":158}
I20250626 14:07:02.658463 3650 maintenance_manager.cc:419] P 24ad911d71b6433f8c7939a1d1f7b782: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 2644 bytes on disk
I20250626 14:07:02.658949 3649 maintenance_manager.cc:643] P 24ad911d71b6433f8c7939a1d1f7b782: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20250626 14:07:02.966468 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 146: this message will repeat every 5th retry.
W20250626 14:07:03.240470 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 146: this message will repeat every 5th retry.
W20250626 14:07:06.093302 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 151: this message will repeat every 5th retry.
W20250626 14:07:06.135077 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 151: this message will repeat every 5th retry.
I20250626 14:07:07.608570 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:07:07.608702 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:07:07.927788 4704 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:07:07.927932 4704 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:07:07.928551 4704 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:07:07.929023 3647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250626 14:07:07.929668 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:07:07.929744 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:07:07.929801 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:07:07.929924 4704 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:07:08.162338 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 151: this message will repeat every 5th retry.
W20250626 14:07:08.212247 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 151: this message will repeat every 5th retry.
W20250626 14:07:10.902272 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 156: this message will repeat every 5th retry.
W20250626 14:07:10.995111 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 156: this message will repeat every 5th retry.
I20250626 14:07:11.778666 4565 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:07:11.778818 4565 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:07:11.779348 4565 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:07:11.780210 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:07:11.780310 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:07:11.780361 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:07:11.780519 4565 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 14:07:12.610713 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:07:12.610827 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:07:13.272450 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 156: this message will repeat every 5th retry.
W20250626 14:07:13.395745 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 156: this message will repeat every 5th retry.
W20250626 14:07:15.854238 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 161: this message will repeat every 5th retry.
W20250626 14:07:16.049032 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 161: this message will repeat every 5th retry.
I20250626 14:07:17.612730 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:07:17.612850 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:07:18.008627 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 161: this message will repeat every 5th retry.
W20250626 14:07:18.101730 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 161: this message will repeat every 5th retry.
W20250626 14:07:20.821189 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 166: this message will repeat every 5th retry.
W20250626 14:07:20.935796 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 166: this message will repeat every 5th retry.
I20250626 14:07:22.614987 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:07:22.615111 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:07:22.627493 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 166: this message will repeat every 5th retry.
W20250626 14:07:23.074705 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 166: this message will repeat every 5th retry.
I20250626 14:07:23.260942 4878 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:07:23.261126 4878 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:07:23.261657 4878 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:07:23.262190 3645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250626 14:07:23.262849 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:07:23.262985 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:07:23.263048 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:07:23.263195 4878 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:07:25.603080 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 171: this message will repeat every 5th retry.
I20250626 14:07:26.260211 4902 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:07:26.260367 4902 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:07:26.260891 4902 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:07:26.261667 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:07:26.261843 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:07:26.261897 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:07:26.262053 4902 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:07:26.371450 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 171: this message will repeat every 5th retry.
I20250626 14:07:27.616997 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:07:27.617125 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:07:28.024245 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 171: this message will repeat every 5th retry.
W20250626 14:07:28.191138 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 171: this message will repeat every 5th retry.
W20250626 14:07:30.607604 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 176: this message will repeat every 5th retry.
W20250626 14:07:31.377965 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 176: this message will repeat every 5th retry.
W20250626 14:07:32.413724 1726 heartbeater.cc:646] Failed to heartbeat to 127.1.11.253:44447 (611 consecutive failures): Network error: Failed to ping master at 127.1.11.253:44447: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:07:32.618950 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:07:32.619062 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:07:32.974259 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 176: this message will repeat every 5th retry.
W20250626 14:07:33.100143 1861 heartbeater.cc:646] Failed to heartbeat to 127.1.11.253:44447 (614 consecutive failures): Network error: Failed to ping master at 127.1.11.253:44447: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:07:33.218495 1996 heartbeater.cc:646] Failed to heartbeat to 127.1.11.253:44447 (299 consecutive failures): Network error: Failed to ping master at 127.1.11.253:44447: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:07:33.276598 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 176: this message will repeat every 5th retry.
W20250626 14:07:35.721719 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 181: this message will repeat every 5th retry.
W20250626 14:07:36.488910 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 181: this message will repeat every 5th retry.
I20250626 14:07:37.617015 4781 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:07:37.617156 4781 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:07:37.617633 4781 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:07:37.618405 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:07:37.618537 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:07:37.618583 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:07:37.618780 4781 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 14:07:37.620978 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:07:37.621086 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:07:37.819885 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 181: this message will repeat every 5th retry.
W20250626 14:07:38.035064 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 181: this message will repeat every 5th retry.
I20250626 14:07:39.865478 5047 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:07:39.865646 5047 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:07:39.866200 5047 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:07:39.866889 3647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250626 14:07:39.867539 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:07:39.867658 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:07:39.867741 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:07:39.867892 5047 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:07:40.735271 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 186: this message will repeat every 5th retry.
W20250626 14:07:41.718577 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 186: this message will repeat every 5th retry.
I20250626 14:07:42.333828 5074 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:07:42.333971 5074 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:07:42.334362 5074 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:07:42.335546 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:07:42.335647 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:07:42.335734 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:07:42.335902 5074 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250626 14:07:42.623464 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:07:42.623606 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:07:42.721359 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 186: this message will repeat every 5th retry.
W20250626 14:07:42.795775 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 186: this message will repeat every 5th retry.
I20250626 14:07:43.817155 5096 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:07:43.817411 5096 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:07:43.818338 5096 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:07:43.820227 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:07:43.820374 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:07:43.820423 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:07:43.820585 5099 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:07:45.693329 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 191: this message will repeat every 5th retry.
W20250626 14:07:47.160706 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 191: this message will repeat every 5th retry.
I20250626 14:07:47.625442 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:07:47.625548 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:07:47.750491 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 191: this message will repeat every 5th retry.
W20250626 14:07:48.070498 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 191: this message will repeat every 5th retry.
W20250626 14:07:50.393244 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 196: this message will repeat every 5th retry.
W20250626 14:07:52.306507 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 196: this message will repeat every 5th retry.
W20250626 14:07:52.512394 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 196: this message will repeat every 5th retry.
I20250626 14:07:52.627802 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:07:52.627939 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:07:52.728730 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 196: this message will repeat every 5th retry.
I20250626 14:07:53.420744 5204 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:07:53.420887 5204 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:07:53.421319 5204 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:07:53.421845 3645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250626 14:07:53.422518 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:07:53.422601 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:07:53.422684 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:07:53.422873 5204 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:07:55.588146 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 201: this message will repeat every 5th retry.
W20250626 14:07:57.519320 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 201: this message will repeat every 5th retry.
I20250626 14:07:57.635025 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:07:57.635157 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:07:57.669778 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 201: this message will repeat every 5th retry.
W20250626 14:07:57.809381 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 201: this message will repeat every 5th retry.
W20250626 14:08:00.482400 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 206: this message will repeat every 5th retry.
I20250626 14:08:01.282748 5099 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:08:01.282895 5099 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:08:01.283448 5099 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:08:01.284297 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:08:01.284622 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:08:01.284695 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:08:01.285121 5099 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:08:02.105144 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 206: this message will repeat every 5th retry.
W20250626 14:08:02.618340 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 206: this message will repeat every 5th retry.
I20250626 14:08:02.637352 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:08:02.637475 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:08:02.755687 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 206: this message will repeat every 5th retry.
I20250626 14:08:03.857494 5161 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:08:03.857642 5161 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:08:03.858086 5161 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:08:03.858870 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:08:03.859009 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:08:03.859062 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:08:03.859228 5161 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:08:05.471201 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 211: this message will repeat every 5th retry.
W20250626 14:08:06.784993 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 211: this message will repeat every 5th retry.
W20250626 14:08:07.432463 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 211: this message will repeat every 5th retry.
W20250626 14:08:07.564386 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 211: this message will repeat every 5th retry.
I20250626 14:08:07.639623 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:08:07.639743 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:08:07.753139 5344 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:08:07.753288 5344 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:08:07.753736 5344 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:08:07.754237 3645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250626 14:08:07.754971 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:08:07.755048 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:08:07.755106 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:08:07.755235 5344 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:08:10.081902 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 216: this message will repeat every 5th retry.
W20250626 14:08:11.577431 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 216: this message will repeat every 5th retry.
W20250626 14:08:12.040845 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 216: this message will repeat every 5th retry.
I20250626 14:08:12.643944 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:08:12.644068 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:08:12.750460 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 216: this message will repeat every 5th retry.
W20250626 14:08:15.297524 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 221: this message will repeat every 5th retry.
W20250626 14:08:16.747956 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 221: this message will repeat every 5th retry.
W20250626 14:08:17.297312 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 221: this message will repeat every 5th retry.
W20250626 14:08:17.571659 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 221: this message will repeat every 5th retry.
I20250626 14:08:17.646139 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:08:17.646258 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:08:19.313848 5376 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:08:19.313997 5376 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:08:19.314522 5376 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:08:19.315295 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:08:19.315629 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:08:19.315696 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:08:19.315866 5376 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:08:19.621739 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 226: this message will repeat every 5th retry.
W20250626 14:08:21.674312 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 226: this message will repeat every 5th retry.
W20250626 14:08:21.778038 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 226: this message will repeat every 5th retry.
I20250626 14:08:22.648180 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:08:22.648295 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:08:22.767957 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 226: this message will repeat every 5th retry.
I20250626 14:08:23.314627 5505 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:08:23.314754 5505 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:08:23.315279 5505 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:08:23.315860 3645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250626 14:08:23.316591 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:08:23.316684 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:08:23.316740 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:08:23.316862 5505 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:08:24.658151 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 231: this message will repeat every 5th retry.
W20250626 14:08:26.147136 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 231: this message will repeat every 5th retry.
W20250626 14:08:26.738397 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 231: this message will repeat every 5th retry.
W20250626 14:08:27.304651 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 231: this message will repeat every 5th retry.
I20250626 14:08:27.650218 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:08:27.650322 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:08:30.072893 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 236: this message will repeat every 5th retry.
W20250626 14:08:31.322134 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 236: this message will repeat every 5th retry.
W20250626 14:08:32.211400 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 236: this message will repeat every 5th retry.
W20250626 14:08:32.555609 1726 heartbeater.cc:646] Failed to heartbeat to 127.1.11.253:44447 (859 consecutive failures): Network error: Failed to ping master at 127.1.11.253:44447: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:08:32.555645 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 236: this message will repeat every 5th retry.
I20250626 14:08:32.652240 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:08:32.652359 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:08:33.168045 1861 heartbeater.cc:646] Failed to heartbeat to 127.1.11.253:44447 (869 consecutive failures): Network error: Failed to ping master at 127.1.11.253:44447: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:08:33.342521 1996 heartbeater.cc:646] Failed to heartbeat to 127.1.11.253:44447 (419 consecutive failures): Network error: Failed to ping master at 127.1.11.253:44447: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:08:34.593108 5511 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:08:34.593245 5511 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:08:34.593696 5511 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:08:34.594305 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:08:34.594432 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:08:34.594480 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:08:34.594604 5511 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 14:08:34.707288 5621 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:08:34.707427 5621 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:08:34.707891 5621 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:08:34.708391 3647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250626 14:08:34.709003 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:08:34.709084 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:08:34.709174 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:08:34.709314 5621 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:08:34.950934 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 241: this message will repeat every 5th retry.
I20250626 14:08:35.425987 5634 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:08:35.426131 5634 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:08:35.426601 5634 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:08:35.427861 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:08:35.427961 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:08:35.428045 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:08:35.428205 5634 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250626 14:08:35.968910 5640 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:08:35.969070 5640 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:08:35.969578 5640 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:08:35.970815 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:08:35.970913 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:08:35.970999 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:08:35.971206 5640 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:08:36.292212 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 241: this message will repeat every 5th retry.
W20250626 14:08:37.427362 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 241: this message will repeat every 5th retry.
W20250626 14:08:37.495239 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 241: this message will repeat every 5th retry.
I20250626 14:08:37.654230 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:08:37.654348 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:08:39.603060 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 246: this message will repeat every 5th retry.
I20250626 14:08:40.815156 5686 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:08:40.815312 5686 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:08:40.815780 5686 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:08:40.816720 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:08:40.816907 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:08:40.816984 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:08:40.817139 5686 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:08:41.676481 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 246: this message will repeat every 5th retry.
W20250626 14:08:42.130863 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 246: this message will repeat every 5th retry.
W20250626 14:08:42.566686 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 246: this message will repeat every 5th retry.
I20250626 14:08:42.656564 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:08:42.656687 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:08:43.974929 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 251: this message will repeat every 5th retry.
I20250626 14:08:44.718779 5661 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:08:44.718936 5661 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:08:44.719381 5661 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:08:44.719964 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:08:44.720094 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:08:44.720152 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:08:44.720310 5661 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:08:46.650846 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 251: this message will repeat every 5th retry.
W20250626 14:08:47.380817 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 251: this message will repeat every 5th retry.
I20250626 14:08:47.658643 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:08:47.658758 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:08:48.063638 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 251: this message will repeat every 5th retry.
W20250626 14:08:49.144484 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 256: this message will repeat every 5th retry.
I20250626 14:08:49.191354 5661 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:08:49.191504 5661 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:08:49.191927 5661 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:08:49.192747 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:08:49.192860 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:08:49.192912 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:08:49.193074 5661 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 14:08:51.423358 5795 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:08:51.423506 5795 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:08:51.424116 5795 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:08:51.424759 3645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250626 14:08:51.425500 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:08:51.425597 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:08:51.425689 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:08:51.425846 5795 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:08:52.266196 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 256: this message will repeat every 5th retry.
W20250626 14:08:52.509366 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 256: this message will repeat every 5th retry.
I20250626 14:08:52.660785 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:08:52.660897 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:08:53.044989 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 256: this message will repeat every 5th retry.
W20250626 14:08:54.248988 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 261: this message will repeat every 5th retry.
I20250626 14:08:55.091158 5816 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:08:55.091312 5816 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:08:55.091763 5816 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:08:55.092557 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:08:55.092689 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:08:55.092738 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:08:55.092952 5816 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:08:57.116370 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 261: this message will repeat every 5th retry.
W20250626 14:08:57.242079 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 261: this message will repeat every 5th retry.
I20250626 14:08:57.663218 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:08:57.663376 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:08:57.853170 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 261: this message will repeat every 5th retry.
W20250626 14:08:59.159749 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 266: this message will repeat every 5th retry.
W20250626 14:09:01.984246 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 266: this message will repeat every 5th retry.
W20250626 14:09:02.152988 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 266: this message will repeat every 5th retry.
I20250626 14:09:02.665752 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:09:02.665863 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:09:03.222227 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 266: this message will repeat every 5th retry.
W20250626 14:09:04.188858 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 271: this message will repeat every 5th retry.
I20250626 14:09:06.304414 5939 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:09:06.304580 5939 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:09:06.305125 5939 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:09:06.305713 3647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250626 14:09:06.306490 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:09:06.306627 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:09:06.306699 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:09:06.306869 5939 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:09:06.796828 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 271: this message will repeat every 5th retry.
W20250626 14:09:07.052778 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 271: this message will repeat every 5th retry.
I20250626 14:09:07.667930 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:09:07.668044 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:09:07.854038 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 271: this message will repeat every 5th retry.
W20250626 14:09:09.283667 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 276: this message will repeat every 5th retry.
I20250626 14:09:11.097584 5816 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:09:11.097743 5816 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:09:11.098309 5816 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:09:11.099133 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:09:11.099269 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:09:11.099326 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:09:11.099505 5816 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:09:11.620633 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 276: this message will repeat every 5th retry.
W20250626 14:09:12.073295 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 276: this message will repeat every 5th retry.
I20250626 14:09:12.670090 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:09:12.670202 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:09:12.924189 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 276: this message will repeat every 5th retry.
I20250626 14:09:13.992980 6023 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:09:13.993111 6023 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:09:13.993523 6023 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:09:13.994050 3647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250626 14:09:13.994625 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:09:13.994751 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:09:13.994823 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:09:13.994953 6023 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:09:14.146402 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 281: this message will repeat every 5th retry.
I20250626 14:09:14.181063 6014 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:09:14.181207 6014 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:09:14.181670 6014 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:09:14.182422 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:09:14.182535 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:09:14.182590 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:09:14.182737 6014 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 14:09:16.208603 6013 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:09:16.208782 6013 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:09:16.209168 6013 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:09:16.209975 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:09:16.210098 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:09:16.210153 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:09:16.210323 6013 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:09:16.377476 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 281: this message will repeat every 5th retry.
W20250626 14:09:16.638746 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 281: this message will repeat every 5th retry.
I20250626 14:09:17.671945 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:09:17.672065 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:09:18.200248 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 281: this message will repeat every 5th retry.
W20250626 14:09:19.292968 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 286: this message will repeat every 5th retry.
I20250626 14:09:19.650523 6076 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:09:19.650661 6076 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:09:19.651120 6076 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:09:19.651786 3645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250626 14:09:19.652429 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:09:19.652562 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:09:19.652622 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:09:19.652772 6076 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:09:21.413255 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 286: this message will repeat every 5th retry.
W20250626 14:09:21.512998 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 286: this message will repeat every 5th retry.
I20250626 14:09:22.674140 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:09:22.674264 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:09:22.917419 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 286: this message will repeat every 5th retry.
W20250626 14:09:24.420141 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 291: this message will repeat every 5th retry.
W20250626 14:09:26.115259 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 291: this message will repeat every 5th retry.
I20250626 14:09:26.783355 6145 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:09:26.783574 6145 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:09:26.784106 6145 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:09:26.784667 3647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250626 14:09:26.785403 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:09:26.785480 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:09:26.785540 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:09:26.785667 6145 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:09:26.966709 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 291: this message will repeat every 5th retry.
I20250626 14:09:27.676427 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:09:27.676576 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:09:27.691835 6161 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:09:27.692023 6161 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:09:27.692597 6161 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:09:27.693845 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:09:27.693941 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:09:27.694015 3647 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:09:27.694156 6161 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:09:27.775244 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 291: this message will repeat every 5th retry.
W20250626 14:09:29.574982 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 296: this message will repeat every 5th retry.
W20250626 14:09:30.732645 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 296: this message will repeat every 5th retry.
W20250626 14:09:31.992511 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 296: this message will repeat every 5th retry.
W20250626 14:09:32.624614 1726 heartbeater.cc:646] Failed to heartbeat to 127.1.11.253:44447 (1105 consecutive failures): Network error: Failed to ping master at 127.1.11.253:44447: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
I20250626 14:09:32.678709 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:09:32.678822 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:09:32.722764 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 296: this message will repeat every 5th retry.
W20250626 14:09:33.330258 1859 heartbeater.cc:646] Failed to heartbeat to 127.1.11.252:37925 (1134 consecutive failures): Network error: Failed to ping master at 127.1.11.252:37925: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:09:33.463666 1994 heartbeater.cc:646] Failed to heartbeat to 127.1.11.252:37925 (551 consecutive failures): Network error: Failed to ping master at 127.1.11.252:37925: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
W20250626 14:09:34.749789 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 301: this message will repeat every 5th retry.
I20250626 14:09:36.035735 6228 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:09:36.035910 6228 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:09:36.036584 6228 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:09:36.037335 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:09:36.037464 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:09:36.037503 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:09:36.037631 6228 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:09:36.086108 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 301: this message will repeat every 5th retry.
I20250626 14:09:36.380358 3650 maintenance_manager.cc:419] P 24ad911d71b6433f8c7939a1d1f7b782: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 2644 bytes on disk
I20250626 14:09:36.383153 3649 maintenance_manager.cc:643] P 24ad911d71b6433f8c7939a1d1f7b782: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.001s sys 0.001s Metrics: {"dirs.queue_time_us":162,"thread_start_us":157,"threads_started":2,"wal-append.queue_time_us":163}
W20250626 14:09:37.028774 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 301: this message will repeat every 5th retry.
W20250626 14:09:37.258054 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 301: this message will repeat every 5th retry.
I20250626 14:09:37.680444 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:09:37.680605 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:09:39.601464 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 306: this message will repeat every 5th retry.
W20250626 14:09:41.319164 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 306: this message will repeat every 5th retry.
W20250626 14:09:42.178758 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 306: this message will repeat every 5th retry.
W20250626 14:09:42.519196 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 306: this message will repeat every 5th retry.
I20250626 14:09:42.682610 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:09:42.682718 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250626 14:09:43.197094 6314 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:09:43.197237 6314 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:09:43.197795 6314 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:09:43.198338 3645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250626 14:09:43.198988 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:09:43.199075 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:09:43.199165 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:09:43.199319 6314 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:09:44.131707 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 311: this message will repeat every 5th retry.
W20250626 14:09:46.496748 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 311: this message will repeat every 5th retry.
W20250626 14:09:47.085140 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 311: this message will repeat every 5th retry.
W20250626 14:09:47.525143 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 311: this message will repeat every 5th retry.
I20250626 14:09:47.684584 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:09:47.684691 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:09:48.971351 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 316: this message will repeat every 5th retry.
I20250626 14:09:50.261706 6350 raft_consensus.cc:491] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e517893e13f546c0b40d976633b49651)
I20250626 14:09:50.261855 6350 raft_consensus.cc:513] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e517893e13f546c0b40d976633b49651" member_type: VOTER last_known_addr { host: "127.1.11.194" port: 44397 } } peers { permanent_uuid: "0c8941c4e30348ed9bc4a3262cd9fe99" member_type: VOTER last_known_addr { host: "127.1.11.195" port: 43231 } } peers { permanent_uuid: "4e57adb83e9f451488ed1d0034043e7b" member_type: VOTER last_known_addr { host: "127.1.11.193" port: 35671 } }
I20250626 14:09:50.262306 6350 leader_election.cc:290] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e517893e13f546c0b40d976633b49651 (127.1.11.194:44397), 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231)
W20250626 14:09:50.263270 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e517893e13f546c0b40d976633b49651 (127.1.11.194:44397): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
W20250626 14:09:50.263429 1613 leader_election.cc:343] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Not found: Tablet not found: a4e9c6ce944b4ec6af44f7852cac93d3
I20250626 14:09:50.263485 1613 leader_election.cc:304] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e57adb83e9f451488ed1d0034043e7b; no voters: 0c8941c4e30348ed9bc4a3262cd9fe99, e517893e13f546c0b40d976633b49651
I20250626 14:09:50.263608 6350 raft_consensus.cc:2747] T a4e9c6ce944b4ec6af44f7852cac93d3 P 4e57adb83e9f451488ed1d0034043e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 14:09:51.315398 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 316: this message will repeat every 5th retry.
W20250626 14:09:52.274020 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 316: this message will repeat every 5th retry.
I20250626 14:09:52.686586 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:09:52.686702 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:09:52.937481 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 316: this message will repeat every 5th retry.
W20250626 14:09:53.931676 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 321: this message will repeat every 5th retry.
I20250626 14:09:55.729300 6447 raft_consensus.cc:491] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:09:55.729467 6447 raft_consensus.cc:513] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ad911d71b6433f8c7939a1d1f7b782" member_type: VOTER last_known_addr { host: "127.1.11.254" port: 36707 } } peers { permanent_uuid: "07532b7648e4414789ed2b099bad2b19" member_type: VOTER last_known_addr { host: "127.1.11.253" port: 44447 } } peers { permanent_uuid: "81db61eed5584e7e8b0cf40e5441f88e" member_type: VOTER last_known_addr { host: "127.1.11.252" port: 37925 } }
I20250626 14:09:55.729897 6447 leader_election.cc:290] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447), 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925)
W20250626 14:09:55.730465 3645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250626 14:09:55.731036 3647 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 07532b7648e4414789ed2b099bad2b19 (127.1.11.253:44447): Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:09:55.731118 3645 leader_election.cc:336] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 81db61eed5584e7e8b0cf40e5441f88e (127.1.11.252:37925): Network error: Client connection negotiation failed: client connection to 127.1.11.252:37925: connect: Connection refused (error 111)
I20250626 14:09:55.731184 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ad911d71b6433f8c7939a1d1f7b782; no voters: 07532b7648e4414789ed2b099bad2b19, 81db61eed5584e7e8b0cf40e5441f88e
I20250626 14:09:55.731308 6447 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 24ad911d71b6433f8c7939a1d1f7b782 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250626 14:09:56.212814 1613 consensus_peers.cc:487] T d21f45117538444ba7b968411a58f099 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21f45117538444ba7b968411a58f099. This is attempt 321: this message will repeat every 5th retry.
W20250626 14:09:57.585917 1748 consensus_peers.cc:487] T 680301fb5722406194f4dcadeb726964 P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680301fb5722406194f4dcadeb726964. This is attempt 321: this message will repeat every 5th retry.
I20250626 14:09:57.688829 3628 master_runner.cc:186] Error getting master registration for 127.1.11.253:44447: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111),
I20250626 14:09:57.688941 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250626 14:09:58.110770 1613 consensus_peers.cc:487] T b325108f7de949199cc10dbe259a0651 P 4e57adb83e9f451488ed1d0034043e7b -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b325108f7de949199cc10dbe259a0651. This is attempt 321: this message will repeat every 5th retry.
W20250626 14:09:58.314385 1748 consensus_peers.cc:487] T 63672ab3fcd14a8783f3821d924c26bd P e517893e13f546c0b40d976633b49651 -> Peer 0c8941c4e30348ed9bc4a3262cd9fe99 (127.1.11.195:43231): Couldn't send request to peer 0c8941c4e30348ed9bc4a3262cd9fe99. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63672ab3fcd14a8783f3821d924c26bd. This is attempt 326: this message will repeat every 5th retry.
W20250626 14:09:59.934993 2001 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:09:59.935199 2001 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:09:59.935246 2001 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
W20250626 14:09:59.935274 2001 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945847 2023 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Aborted at 1750946999 (unix time) try "date -d @1750946999" if you are using GNU date ***
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000042f) received by PID 1071 (TID 0x7f2f61fa9700) from PID 1071; stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950443 2027 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950443 2027 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951601 2022 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950443 2027 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951601 2022 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951640 2010 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f7012cd4bf5d40d6af93cf9b01405dcf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950443 2027 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951601 2022 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951640 2010 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f7012cd4bf5d40d6af93cf9b01405dcf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951700 2013 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_34cb28e2c8d740aba5c52377533fdfa4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950443 2027 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951601 2022 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951640 2010 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f7012cd4bf5d40d6af93cf9b01405dcf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951700 2013 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_34cb28e2c8d740aba5c52377533fdfa4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951812 2025 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950443 2027 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951601 2022 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951640 2010 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f7012cd4bf5d40d6af93cf9b01405dcf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951700 2013 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_34cb28e2c8d740aba5c52377533fdfa4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951812 2025 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951915 2016 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f98cc740eae3444b88bb688eb8c8a316 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950443 2027 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951601 2022 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951640 2010 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f7012cd4bf5d40d6af93cf9b01405dcf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951700 2013 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_34cb28e2c8d740aba5c52377533fdfa4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951812 2025 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951915 2016 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f98cc740eae3444b88bb688eb8c8a316 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951993 2026 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f2f78474980 (unknown) at ??:0
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950443 2027 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951601 2022 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951640 2010 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f7012cd4bf5d40d6af93cf9b01405dcf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951700 2013 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_34cb28e2c8d740aba5c52377533fdfa4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951812 2025 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951915 2016 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f98cc740eae3444b88bb688eb8c8a316 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951993 2026 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.952883 2028 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950443 2027 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951601 2022 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951640 2010 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f7012cd4bf5d40d6af93cf9b01405dcf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951700 2013 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_34cb28e2c8d740aba5c52377533fdfa4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951812 2025 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951915 2016 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f98cc740eae3444b88bb688eb8c8a316 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951993 2026 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.952883 2028 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.952963 2018 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2c8dee0163784a60a8468b53619247f4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950443 2027 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951601 2022 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951640 2010 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f7012cd4bf5d40d6af93cf9b01405dcf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951700 2013 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_34cb28e2c8d740aba5c52377533fdfa4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951812 2025 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951915 2016 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f98cc740eae3444b88bb688eb8c8a316 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951993 2026 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.952883 2028 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.952963 2018 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2c8dee0163784a60a8468b53619247f4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.953074 2008 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_213d95d0e97546c298d94c45091826cb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950443 2027 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951601 2022 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951640 2010 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f7012cd4bf5d40d6af93cf9b01405dcf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951700 2013 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_34cb28e2c8d740aba5c52377533fdfa4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951812 2025 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951915 2016 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f98cc740eae3444b88bb688eb8c8a316 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951993 2026 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.952883 2028 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.952963 2018 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2c8dee0163784a60a8468b53619247f4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.953074 2008 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_213d95d0e97546c298d94c45091826cb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.954114 2029 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f2f76086fb7 gsignal at ??:0
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950443 2027 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951601 2022 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951640 2010 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f7012cd4bf5d40d6af93cf9b01405dcf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951700 2013 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_34cb28e2c8d740aba5c52377533fdfa4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951812 2025 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951915 2016 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f98cc740eae3444b88bb688eb8c8a316 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951993 2026 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.952883 2028 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.952963 2018 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2c8dee0163784a60a8468b53619247f4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.953074 2008 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_213d95d0e97546c298d94c45091826cb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.954114 2029 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.955283 2012 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8c238e2636e9491f938a48294f961982 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950443 2027 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951601 2022 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951640 2010 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f7012cd4bf5d40d6af93cf9b01405dcf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951700 2013 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_34cb28e2c8d740aba5c52377533fdfa4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951812 2025 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951915 2016 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f98cc740eae3444b88bb688eb8c8a316 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951993 2026 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.952883 2028 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.952963 2018 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2c8dee0163784a60a8468b53619247f4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.953074 2008 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_213d95d0e97546c298d94c45091826cb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.954114 2029 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.955283 2012 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8c238e2636e9491f938a48294f961982 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.955291 2017 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_33ee3521f64d4cefb8856259510e5f0e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250626 14:09:59.945863 2014 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fdd6a46b24984290ac6f503f5684b003 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.945849 2020 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.946971 2032 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.948114 2031 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111) :09:59.
8143 2019 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.949290 2021 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950397 2033 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950438 2009 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_566f55c5d8c34043a37041fef8cb9f45 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950488 2030 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.950443 2027 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951601 2022 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951640 2010 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f7012cd4bf5d40d6af93cf9b01405dcf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951700 2013 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_34cb28e2c8d740aba5c52377533fdfa4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951812 2025 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951915 2016 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f98cc740eae3444b88bb688eb8c8a316 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.951993 2026 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.952883 2028 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.952963 2018 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2c8dee0163784a60a8468b53619247f4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.953074 2008 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_213d95d0e97546c298d94c45091826cb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.954114 2029 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.955283 2012 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8c238e2636e9491f938a48294f961982 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.955291 2017 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_33ee3521f64d4cefb8856259510e5f0e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)F20250626 14:09:59.955400 2015 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5bf40f09daaa4240a8bcab30e1aa341f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.11.254:36707,127.1.11.253:44447,127.1.11.252:37925, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.11.253:44447: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f2f76088921 abort at ??:0
@ 0x7f2f76eefdcd google::LogMessage::Fail() at ??:0
@ 0x7f2f76ef3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f2f76eef7cc google::LogMessage::Flush() at ??:0
@ 0x7f2f76ef0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x5572e709da99 kudu::MasterStressTest::CreateTableThread() at ??:0
@ 0x5572e70b07b0 std::__invoke_impl<>() at ??:0
@ 0x5572e70ae4ec std::__invoke<>() at ??:0
@ 0x5572e70b5b3d _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
@ 0x5572e70b4b41 std::thread::_Invoker<>::operator()() at ??:0
@ 0x5572e70b38be std::thread::_State_impl<>::_M_run() at ??:0
@ 0x7f2f7670e6df (unknown) at ??:0
@ 0x7f2f784696db start_thread at ??:0
@ 0x7f2f7616971f clone at ??:0