Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251025 01:55:02.085034 9718 test_util.cc:276] Using random seed: -196070014
2025-10-25T01:55:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T01:55:02Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasknR9X9f/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasknR9X9f/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/security.properties
2025-10-25 01:55:07: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasknR9X9f/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasknR9X9f/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20251025 01:55:14.650807 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.190:35925
--webserver_interface=127.9.125.190
--webserver_port=0
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251025 01:55:14.939033 9959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:14.939327 9959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:14.939450 9959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 01:55:14.946976 9959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 01:55:14.947052 9959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:14.947110 9959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 01:55:14.947190 9959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 01:55:14.955986 9959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.190:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.125.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:14.957440 9959 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:14.959101 9959 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:14.966614 9965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:14.966658 9964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:14.966619 9967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:14.967300 9959 server_base.cc:1047] running on GCE node
I20251025 01:55:14.968005 9959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:14.969524 9959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:14.970722 9959 hybrid_clock.cc:648] HybridClock initialized: now 1761357314970691 us; error 30 us; skew 500 ppm
I20251025 01:55:14.971022 9959 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:14.972990 9959 webserver.cc:492] Webserver started at http://127.9.125.190:45449/ using document root <none> and password file <none>
I20251025 01:55:14.973608 9959 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:14.973685 9959 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:14.973990 9959 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 01:55:14.975976 9959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/instance:
uuid: "76559300dae44c0d8a17b583a8875b3f"
format_stamp: "Formatted at 2025-10-25 01:55:14 on dist-test-slave-mkd1"
I20251025 01:55:14.976660 9959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal/instance:
uuid: "76559300dae44c0d8a17b583a8875b3f"
format_stamp: "Formatted at 2025-10-25 01:55:14 on dist-test-slave-mkd1"
I20251025 01:55:14.981374 9959 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251025 01:55:14.984227 9973 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:14.985312 9959 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 01:55:14.985450 9959 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
uuid: "76559300dae44c0d8a17b583a8875b3f"
format_stamp: "Formatted at 2025-10-25 01:55:14 on dist-test-slave-mkd1"
I20251025 01:55:14.985635 9959 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:15.029634 9959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:15.030452 9959 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:15.030699 9959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:15.048025 9959 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.190:35925
I20251025 01:55:15.048060 10024 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.190:35925 every 8 connection(s)
I20251025 01:55:15.049305 9959 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
I20251025 01:55:15.052947 10025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:15.054976 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 9959
I20251025 01:55:15.055208 9718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal/instance
I20251025 01:55:15.057505 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.189:45553
--webserver_interface=127.9.125.189
--webserver_port=0
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251025 01:55:15.058306 10025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } has no permanent_uuid. Determining permanent_uuid...
I20251025 01:55:15.077280 10025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } has no permanent_uuid. Determining permanent_uuid...
W20251025 01:55:15.078217 9976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:55:15.079697 10025 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.125.189:45553: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:15.129002 10025 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } attempt: 1
W20251025 01:55:15.130776 10025 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.125.189:45553: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:15.199059 10025 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } attempt: 2
W20251025 01:55:15.200771 10025 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.125.189:45553: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:15.292024 10025 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } attempt: 3
W20251025 01:55:15.293694 10025 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.125.189:45553: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:55:15.344519 10026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:15.344787 10026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:15.344852 10026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 01:55:15.351684 10026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 01:55:15.351755 10026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:15.351801 10026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 01:55:15.351841 10026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 01:55:15.360093 10026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.189:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.125.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:15.361410 10026 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:15.362957 10026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:15.369792 10034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:15.369817 10035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:15.370308 10037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:15.371034 10026 server_base.cc:1047] running on GCE node
I20251025 01:55:15.371657 10026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:15.372789 10026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:15.373960 10026 hybrid_clock.cc:648] HybridClock initialized: now 1761357315373920 us; error 34 us; skew 500 ppm
I20251025 01:55:15.374243 10026 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:15.376020 10026 webserver.cc:492] Webserver started at http://127.9.125.189:40341/ using document root <none> and password file <none>
I20251025 01:55:15.376627 10026 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:15.376709 10026 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:15.377027 10026 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 01:55:15.379050 10026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data/instance:
uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce"
format_stamp: "Formatted at 2025-10-25 01:55:15 on dist-test-slave-mkd1"
I20251025 01:55:15.379720 10026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal/instance:
uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce"
format_stamp: "Formatted at 2025-10-25 01:55:15 on dist-test-slave-mkd1"
I20251025 01:55:15.384224 10026 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20251025 01:55:15.387154 10043 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:15.388267 10026 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 01:55:15.388406 10026 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce"
format_stamp: "Formatted at 2025-10-25 01:55:15 on dist-test-slave-mkd1"
I20251025 01:55:15.388597 10026 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:15.404096 10026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:15.404772 10026 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:15.404994 10026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:15.422674 10026 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.189:45553
I20251025 01:55:15.422701 10094 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.189:45553 every 8 connection(s)
I20251025 01:55:15.423969 10026 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data/info.pb
I20251025 01:55:15.427482 10095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:15.432430 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 10026
I20251025 01:55:15.432600 9718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal/instance
I20251025 01:55:15.432755 10095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } has no permanent_uuid. Determining permanent_uuid...
I20251025 01:55:15.433799 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.188:36641
--webserver_interface=127.9.125.188
--webserver_port=0
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251025 01:55:15.436928 10025 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } attempt: 4
I20251025 01:55:15.445475 10095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } has no permanent_uuid. Determining permanent_uuid...
I20251025 01:55:15.446753 10025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } has no permanent_uuid. Determining permanent_uuid...
W20251025 01:55:15.448601 10025 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.125.188:36641: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:15.450806 10095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } has no permanent_uuid. Determining permanent_uuid...
W20251025 01:55:15.451671 10044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
W20251025 01:55:15.452937 10095 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.125.188:36641: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:15.502224 10095 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } attempt: 1
W20251025 01:55:15.503980 10095 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.125.188:36641: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:15.507790 10025 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } attempt: 1
W20251025 01:55:15.509282 10025 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.125.188:36641: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:15.572217 10095 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } attempt: 2
W20251025 01:55:15.573858 10095 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.125.188:36641: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:15.576463 10025 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } attempt: 2
W20251025 01:55:15.578114 10025 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.125.188:36641: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:15.665118 10095 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } attempt: 3
W20251025 01:55:15.666989 10095 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.125.188:36641: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:15.678313 10025 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } attempt: 3
W20251025 01:55:15.680011 10025 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.125.188:36641: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
W20251025 01:55:15.728150 10097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:15.728417 10097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:15.728542 10097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 01:55:15.735831 10097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 01:55:15.735898 10097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:15.735955 10097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 01:55:15.736011 10097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 01:55:15.744257 10097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.188:36641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.125.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:15.745640 10097 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:15.747141 10097 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:15.754106 10105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:15.754160 10104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:15.754782 10107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:15.755136 10097 server_base.cc:1047] running on GCE node
I20251025 01:55:15.755796 10097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:15.756922 10097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:15.758123 10097 hybrid_clock.cc:648] HybridClock initialized: now 1761357315758080 us; error 47 us; skew 500 ppm
I20251025 01:55:15.758445 10097 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:15.760411 10097 webserver.cc:492] Webserver started at http://127.9.125.188:33371/ using document root <none> and password file <none>
I20251025 01:55:15.761050 10097 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:15.761132 10097 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:15.761457 10097 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 01:55:15.763470 10097 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/instance:
uuid: "286b4c3025e24a7c8107763fa40d3ab1"
format_stamp: "Formatted at 2025-10-25 01:55:15 on dist-test-slave-mkd1"
I20251025 01:55:15.764122 10097 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal/instance:
uuid: "286b4c3025e24a7c8107763fa40d3ab1"
format_stamp: "Formatted at 2025-10-25 01:55:15 on dist-test-slave-mkd1"
I20251025 01:55:15.768693 10097 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20251025 01:55:15.771448 10113 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:15.772490 10097 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 01:55:15.772626 10097 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
uuid: "286b4c3025e24a7c8107763fa40d3ab1"
format_stamp: "Formatted at 2025-10-25 01:55:15 on dist-test-slave-mkd1"
I20251025 01:55:15.772795 10097 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:15.798620 10097 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:15.799358 10097 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:15.799582 10097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:15.810245 10095 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } attempt: 4
W20251025 01:55:15.812325 10095 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.9.125.188:36641: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:15.817749 10097 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.188:36641
I20251025 01:55:15.817776 10164 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.188:36641 every 8 connection(s)
I20251025 01:55:15.819025 10097 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/info.pb
I20251025 01:55:15.821296 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 10097
I20251025 01:55:15.821468 9718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal/instance
I20251025 01:55:15.822768 10165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:15.829784 10165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } has no permanent_uuid. Determining permanent_uuid...
I20251025 01:55:15.840181 10165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } has no permanent_uuid. Determining permanent_uuid...
I20251025 01:55:15.846779 10165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } has no permanent_uuid. Determining permanent_uuid...
I20251025 01:55:15.850212 10025 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } attempt: 4
I20251025 01:55:15.858305 10165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Bootstrap starting.
I20251025 01:55:15.860903 10025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Bootstrap starting.
I20251025 01:55:15.861286 10165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:15.862181 10165 log.cc:826] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:15.863363 10025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:15.864260 10025 log.cc:826] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:15.864343 10165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: No bootstrap required, opened a new log
I20251025 01:55:15.869184 10165 raft_consensus.cc:359] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:15.869413 10165 raft_consensus.cc:385] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:15.869536 10165 raft_consensus.cc:740] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286b4c3025e24a7c8107763fa40d3ab1, State: Initialized, Role: FOLLOWER
I20251025 01:55:15.870051 10165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:15.870795 10175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:15.871055 10175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:15.871433 10025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: No bootstrap required, opened a new log
I20251025 01:55:15.871570 10165 sys_catalog.cc:565] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 01:55:15.876621 10025 raft_consensus.cc:359] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:15.876823 10025 raft_consensus.cc:385] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:15.876921 10025 raft_consensus.cc:740] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76559300dae44c0d8a17b583a8875b3f, State: Initialized, Role: FOLLOWER
I20251025 01:55:15.877555 10025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:15.878556 10190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:15.878886 10190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:15.879663 10025 sys_catalog.cc:565] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: configured and running, proceeding with master startup.
I20251025 01:55:15.884137 10189 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20251025 01:55:15.892057 10205 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 01:55:15.892256 10205 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 01:55:15.888104 10097 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: OK, instance_id { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" instance_seqno: 1761357315043471 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251025 01:55:15.895668 10097 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:15.895812 9959 master_runner.cc:186] Error getting master registration for 127.9.125.189:45553: OK, instance_id { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" instance_seqno: 1761357315417946 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251025 01:55:15.901737 9959 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:15.906349 10175 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:15.906581 10175 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:15.908123 10175 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
I20251025 01:55:15.908921 10000 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "286b4c3025e24a7c8107763fa40d3ab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76559300dae44c0d8a17b583a8875b3f" is_pre_election: true
I20251025 01:55:15.909369 10000 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 286b4c3025e24a7c8107763fa40d3ab1 in term 0.
I20251025 01:55:15.909883 10116 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f; no voters:
I20251025 01:55:15.910274 10175 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:15.910398 10175 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:15.910477 10175 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:15.913455 10175 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:15.914366 10175 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 1 election: Requested vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
I20251025 01:55:15.914404 10000 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "286b4c3025e24a7c8107763fa40d3ab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76559300dae44c0d8a17b583a8875b3f"
I20251025 01:55:15.914589 10000 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:15.917784 10000 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 286b4c3025e24a7c8107763fa40d3ab1 in term 1.
I20251025 01:55:15.918248 10116 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f; no voters:
I20251025 01:55:15.918534 10175 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:15.938992 10175 raft_consensus.cc:697] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 1 LEADER]: Becoming Leader. State: Replica: 286b4c3025e24a7c8107763fa40d3ab1, State: Running, Role: LEADER
I20251025 01:55:15.939656 10175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:15.952636 10208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286b4c3025e24a7c8107763fa40d3ab1. Latest consensus state: current_term: 1 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:15.952942 10208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:15.956323 10210 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 01:55:15.969290 10210 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 01:55:16.004731 10000 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 1 FOLLOWER]: Refusing update from remote peer 286b4c3025e24a7c8107763fa40d3ab1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 01:55:16.005761 10175 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:16.032398 10212 mvcc.cc:204] Tried to move back new op lower bound from 7214519566316781568 to 7214519566094475264. Current Snapshot: MvccSnapshot[applied={T|T < 7214519566316781568}]
W20251025 01:55:16.032996 9979 tablet.cc:2378] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 01:55:16.033160 10190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286b4c3025e24a7c8107763fa40d3ab1. Latest consensus state: current_term: 1 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:16.033360 10190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:16.034890 10190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:16.035044 10190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:16.045887 10208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:16.046126 10208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:16.047179 10213 mvcc.cc:204] Tried to move back new op lower bound from 7214519566316781568 to 7214519566094475264. Current Snapshot: MvccSnapshot[applied={T|T < 7214519566316781568}]
I20251025 01:55:16.048946 10210 catalog_manager.cc:1357] Generated new cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:16.049039 10210 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 01:55:16.070736 10210 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 01:55:16.073699 10210 catalog_manager.cc:1514] Loading token signing keys...
I20251025 01:55:16.111495 10095 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } attempt: 5
I20251025 01:55:16.135056 10210 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Generated new TSK 0
I20251025 01:55:16.136482 10095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Bootstrap starting.
I20251025 01:55:16.137213 10210 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 01:55:16.140158 10210 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251025 01:55:16.141044 10095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:16.142274 10095 log.cc:826] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:16.142433 10210 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251025 01:55:16.151079 10095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: No bootstrap required, opened a new log
I20251025 01:55:16.161306 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.125.129:0
--local_ip_for_outbound_sockets=127.9.125.129
--webserver_interface=127.9.125.129
--webserver_port=0
--tserver_master_addrs=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251025 01:55:16.176075 10095 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:16.176481 10095 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:16.176632 10095 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, State: Initialized, Role: FOLLOWER
I20251025 01:55:16.177529 10095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:16.188109 10219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:16.190191 10219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:16.190124 10069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "286b4c3025e24a7c8107763fa40d3ab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce"
I20251025 01:55:16.190068 10095 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: configured and running, proceeding with master startup.
I20251025 01:55:16.190551 10069 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:16.193881 10069 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 286b4c3025e24a7c8107763fa40d3ab1 in term 1.
I20251025 01:55:16.194283 10070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "286b4c3025e24a7c8107763fa40d3ab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" is_pre_election: true
I20251025 01:55:16.194509 10070 raft_consensus.cc:2376] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 286b4c3025e24a7c8107763fa40d3ab1 in term 1. Re-sending same reply.
I20251025 01:55:16.195616 10068 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Refusing update from remote peer 286b4c3025e24a7c8107763fa40d3ab1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 01:55:16.196478 10208 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:16.251288 10228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:16.251602 10228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:16.251932 10219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286b4c3025e24a7c8107763fa40d3ab1. Latest consensus state: current_term: 1 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:16.252080 10219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:16.287444 10239 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20251025 01:55:16.592351 10218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:16.592592 10218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:16.592712 10218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 01:55:16.599771 10218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:16.599933 10218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.125.129
I20251025 01:55:16.607872 10218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.125.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.125.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.125.129
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:16.609278 10218 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:16.610862 10218 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:16.619508 10245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:16.619462 10244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:16.622462 10218 server_base.cc:1047] running on GCE node
W20251025 01:55:16.622713 10247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:16.623355 10218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:16.624204 10218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:16.625434 10218 hybrid_clock.cc:648] HybridClock initialized: now 1761357316625376 us; error 40 us; skew 500 ppm
I20251025 01:55:16.625733 10218 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:16.627676 10218 webserver.cc:492] Webserver started at http://127.9.125.129:43715/ using document root <none> and password file <none>
I20251025 01:55:16.628296 10218 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:16.628389 10218 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:16.628713 10218 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 01:55:16.630793 10218 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/data/instance:
uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
format_stamp: "Formatted at 2025-10-25 01:55:16 on dist-test-slave-mkd1"
I20251025 01:55:16.631516 10218 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/instance:
uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
format_stamp: "Formatted at 2025-10-25 01:55:16 on dist-test-slave-mkd1"
I20251025 01:55:16.636061 10218 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20251025 01:55:16.638909 10253 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:16.640017 10218 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20251025 01:55:16.640148 10218 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal
uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
format_stamp: "Formatted at 2025-10-25 01:55:16 on dist-test-slave-mkd1"
I20251025 01:55:16.640305 10218 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:16.688189 10218 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:16.688987 10218 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:16.689220 10218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:16.690069 10218 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 01:55:16.692092 10218 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 01:55:16.692170 10218 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:16.692234 10218 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 01:55:16.692299 10218 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:16.699400 10219 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 286b4c3025e24a7c8107763fa40d3ab1)
I20251025 01:55:16.699594 10219 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:16.700613 10219 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:16.701225 10000 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "76559300dae44c0d8a17b583a8875b3f" is_pre_election: true
I20251025 01:55:16.701349 10140 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "286b4c3025e24a7c8107763fa40d3ab1" is_pre_election: true
I20251025 01:55:16.702214 10044 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce; no voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:16.702648 10219 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:55:16.715198 10208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:16.715503 10208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:16.726908 10218 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.129:44581
I20251025 01:55:16.727030 10368 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.129:44581 every 8 connection(s)
I20251025 01:55:16.728336 10218 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/data/info.pb
I20251025 01:55:16.733583 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 10218
I20251025 01:55:16.733793 9718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/instance
I20251025 01:55:16.739462 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.125.130:0
--local_ip_for_outbound_sockets=127.9.125.130
--webserver_interface=127.9.125.130
--webserver_port=0
--tserver_master_addrs=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251025 01:55:16.747641 10371 heartbeater.cc:344] Connected to a master server at 127.9.125.189:45553
I20251025 01:55:16.747983 10371 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:16.748848 10371 heartbeater.cc:507] Master 127.9.125.189:45553 requested a full tablet report, sending...
I20251025 01:55:16.750463 10369 heartbeater.cc:344] Connected to a master server at 127.9.125.188:36641
I20251025 01:55:16.750594 10369 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:16.751121 10060 ts_manager.cc:194] Registered new tserver with Master: c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:16.751179 10369 heartbeater.cc:507] Master 127.9.125.188:36641 requested a full tablet report, sending...
I20251025 01:55:16.752741 10130 ts_manager.cc:194] Registered new tserver with Master: c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:16.754524 10370 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:16.754638 10370 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:16.754658 10130 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.125.129:40173
I20251025 01:55:16.755074 10370 heartbeater.cc:507] Master 127.9.125.190:35925 requested a full tablet report, sending...
I20251025 01:55:16.756815 9990 ts_manager.cc:194] Registered new tserver with Master: c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:16.894006 10205 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:16.894097 10205 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: loading cluster ID for follower catalog manager: success
I20251025 01:55:16.896360 10205 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: acquiring CA information for follower catalog manager: success
I20251025 01:55:16.898367 10205 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 01:55:17.029186 10380 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 286b4c3025e24a7c8107763fa40d3ab1)
I20251025 01:55:17.029366 10380 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:17.030529 10380 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:17.031132 10140 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "286b4c3025e24a7c8107763fa40d3ab1" is_pre_election: true
I20251025 01:55:17.031132 10068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" is_pre_election: true
I20251025 01:55:17.031817 9974 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:17.032166 10380 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 01:55:17.041682 10379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:17.041922 10379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:17.042060 10379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 01:55:17.050288 10379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:17.050479 10379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.125.130
I20251025 01:55:17.064419 10379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.125.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.125.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.125.130
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:17.066210 10379 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:17.068253 10379 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:17.078064 10386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:17.078020 10385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:17.078830 10388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:17.079546 10379 server_base.cc:1047] running on GCE node
I20251025 01:55:17.080086 10379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:17.080910 10379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:17.082115 10379 hybrid_clock.cc:648] HybridClock initialized: now 1761357317082072 us; error 33 us; skew 500 ppm
I20251025 01:55:17.082383 10379 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:17.084435 10379 webserver.cc:492] Webserver started at http://127.9.125.130:36621/ using document root <none> and password file <none>
I20251025 01:55:17.085130 10379 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:17.085234 10379 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:17.085587 10379 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 01:55:17.087913 10379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/data/instance:
uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
format_stamp: "Formatted at 2025-10-25 01:55:17 on dist-test-slave-mkd1"
I20251025 01:55:17.088644 10379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/instance:
uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
format_stamp: "Formatted at 2025-10-25 01:55:17 on dist-test-slave-mkd1"
I20251025 01:55:17.093505 10379 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251025 01:55:17.096557 10394 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:17.097633 10379 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 01:55:17.097776 10379 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal
uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
format_stamp: "Formatted at 2025-10-25 01:55:17 on dist-test-slave-mkd1"
I20251025 01:55:17.097955 10379 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:17.128263 10379 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:17.128969 10379 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:17.129199 10379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:17.130024 10379 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 01:55:17.131554 10379 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 01:55:17.131649 10379 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:17.131747 10379 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 01:55:17.131816 10379 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:17.161054 10379 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.130:43549
I20251025 01:55:17.161113 10506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.130:43549 every 8 connection(s)
I20251025 01:55:17.162300 10379 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/data/info.pb
I20251025 01:55:17.169523 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 10379
I20251025 01:55:17.169723 9718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/instance
I20251025 01:55:17.175390 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.125.131:0
--local_ip_for_outbound_sockets=127.9.125.131
--webserver_interface=127.9.125.131
--webserver_port=0
--tserver_master_addrs=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251025 01:55:17.179056 10507 heartbeater.cc:344] Connected to a master server at 127.9.125.188:36641
I20251025 01:55:17.179090 10509 heartbeater.cc:344] Connected to a master server at 127.9.125.189:45553
I20251025 01:55:17.179430 10509 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:17.179435 10507 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:17.180310 10509 heartbeater.cc:507] Master 127.9.125.189:45553 requested a full tablet report, sending...
I20251025 01:55:17.180335 10507 heartbeater.cc:507] Master 127.9.125.188:36641 requested a full tablet report, sending...
I20251025 01:55:17.181030 10508 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:17.181155 10508 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:17.181576 10508 heartbeater.cc:507] Master 127.9.125.190:35925 requested a full tablet report, sending...
I20251025 01:55:17.181712 10130 ts_manager.cc:194] Registered new tserver with Master: 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:17.181819 10060 ts_manager.cc:194] Registered new tserver with Master: 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:17.182426 9990 ts_manager.cc:194] Registered new tserver with Master: 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:17.182654 10130 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.125.130:40551
I20251025 01:55:17.217226 10515 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 286b4c3025e24a7c8107763fa40d3ab1)
I20251025 01:55:17.217458 10515 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:17.218395 10515 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:17.218863 10000 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "76559300dae44c0d8a17b583a8875b3f" is_pre_election: true
I20251025 01:55:17.218963 10140 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "286b4c3025e24a7c8107763fa40d3ab1" is_pre_election: true
I20251025 01:55:17.219602 10046 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce; no voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:17.219954 10515 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:55:17.256891 10369 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
I20251025 01:55:17.292372 10239 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:17.292497 10239 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: loading cluster ID for follower catalog manager: success
I20251025 01:55:17.295948 10239 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: acquiring CA information for follower catalog manager: success
I20251025 01:55:17.298146 10239 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251025 01:55:17.474555 10514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:17.474850 10514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:17.474988 10514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 01:55:17.482108 10514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:17.482300 10514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.125.131
I20251025 01:55:17.490754 10514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.125.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.125.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.125.131
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:17.492192 10514 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:17.493746 10514 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:17.501837 10522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:17.503101 10521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:17.503525 10524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:17.504875 10514 server_base.cc:1047] running on GCE node
I20251025 01:55:17.505378 10514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:17.506230 10514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:17.507437 10514 hybrid_clock.cc:648] HybridClock initialized: now 1761357317507390 us; error 37 us; skew 500 ppm
I20251025 01:55:17.507707 10514 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:17.509810 10514 webserver.cc:492] Webserver started at http://127.9.125.131:39143/ using document root <none> and password file <none>
I20251025 01:55:17.510421 10514 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:17.510496 10514 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:17.510795 10514 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 01:55:17.512933 10514 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/data/instance:
uuid: "90109d2fdb464942b20891b5856f8ac2"
format_stamp: "Formatted at 2025-10-25 01:55:17 on dist-test-slave-mkd1"
I20251025 01:55:17.513628 10514 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/instance:
uuid: "90109d2fdb464942b20891b5856f8ac2"
format_stamp: "Formatted at 2025-10-25 01:55:17 on dist-test-slave-mkd1"
I20251025 01:55:17.518388 10514 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20251025 01:55:17.521466 10530 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:17.522775 10514 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251025 01:55:17.522907 10514 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal
uuid: "90109d2fdb464942b20891b5856f8ac2"
format_stamp: "Formatted at 2025-10-25 01:55:17 on dist-test-slave-mkd1"
I20251025 01:55:17.523088 10514 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:17.538797 10514 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:17.539520 10514 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:17.539736 10514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:17.540549 10514 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 01:55:17.541971 10514 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 01:55:17.542061 10514 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:17.542158 10514 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 01:55:17.542222 10514 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:17.577821 10514 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.131:41021
I20251025 01:55:17.577854 10642 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.131:41021 every 8 connection(s)
I20251025 01:55:17.579139 10514 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/data/info.pb
I20251025 01:55:17.583541 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 10514
I20251025 01:55:17.583698 9718 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/instance
I20251025 01:55:17.607686 10644 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:17.607976 10644 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:17.608909 10644 heartbeater.cc:507] Master 127.9.125.190:35925 requested a full tablet report, sending...
I20251025 01:55:17.609968 10645 heartbeater.cc:344] Connected to a master server at 127.9.125.189:45553
I20251025 01:55:17.610092 10645 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:17.610559 10645 heartbeater.cc:507] Master 127.9.125.189:45553 requested a full tablet report, sending...
I20251025 01:55:17.611907 10060 ts_manager.cc:194] Registered new tserver with Master: 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:17.612541 9990 ts_manager.cc:194] Registered new tserver with Master: 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:17.618324 10643 heartbeater.cc:344] Connected to a master server at 127.9.125.188:36641
I20251025 01:55:17.618439 10643 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:17.618790 10643 heartbeater.cc:507] Master 127.9.125.188:36641 requested a full tablet report, sending...
I20251025 01:55:17.619622 10130 ts_manager.cc:194] Registered new tserver with Master: 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:17.620615 10130 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.125.131:46983
I20251025 01:55:17.627131 9718 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 01:55:17.665947 10130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33678:
name: "default.Table_aaaf9fef59a24b46a5777425aa3d1c5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:17.666046 10129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33678:
name: "default.Table_437718d96b1e4760a51380e0f777933e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:17.667819 10127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33678:
name: "default.Table_44e4ccc4f2de47198a0599e2577108cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:17.667763 10128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33678:
name: "default.Table_9e3c9d4bfaa242ce9ed8d33fc144f8be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:17.668874 10125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33678:
name: "default.Table_5a6f9a84749b447180f8e258046f31ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:17.668931 10124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33678:
name: "default.Table_39f4a74b6880456585fc71ca55bff11b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:17.669597 10126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33678:
name: "default.Table_19b5f67b2cfa4e62bc0d061b1c2c8ecf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:17.670320 10123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33678:
name: "default.Table_a2d327beb950460cbf080c4c44d89943"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:17.670923 10122 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33678:
name: "default.Table_083845c9d31747a68c31d8a1804a12aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:17.673453 10121 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33678:
name: "default.Table_ebad90bab99347a39c329ef128d4612e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 01:55:17.676482 10130 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_aaaf9fef59a24b46a5777425aa3d1c5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 01:55:17.684474 10507 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
I20251025 01:55:17.707448 9718 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu pid 10097
I20251025 01:55:17.708730 10160 generic_service.cc:196] Checking for leaks (request via RPC)
I20251025 01:55:17.839591 10689 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 286b4c3025e24a7c8107763fa40d3ab1)
I20251025 01:55:17.840003 10689 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:17.841073 10689 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:17.841486 10000 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "76559300dae44c0d8a17b583a8875b3f" is_pre_election: true
I20251025 01:55:17.846807 10140 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "286b4c3025e24a7c8107763fa40d3ab1" is_pre_election: true
I20251025 01:55:17.847471 10044 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce; no voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:17.847800 10689 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:55:17.851742 9718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu with pid 10097
W20251025 01:55:17.878262 10649 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.125.188:36641)
W20251025 01:55:17.880605 10507 heartbeater.cc:646] Failed to heartbeat to 127.9.125.188:36641 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.9.125.188:36641 (error 108)
W20251025 01:55:17.880949 10369 heartbeater.cc:646] Failed to heartbeat to 127.9.125.188:36641 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.9.125.188:36641 (error 108)
W20251025 01:55:17.881227 10045 connection.cc:537] server connection from 127.0.0.1:58642 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:17.881811 10655 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 01:55:17.879972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.125.188:36641 (local address 127.0.0.1:33690)
1025 01:55:17.880399 (+ 427us) negotiation.cc:107] Waiting for socket to connect
1025 01:55:17.880578 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":110}
I20251025 01:55:18.108559 10690 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 286b4c3025e24a7c8107763fa40d3ab1)
I20251025 01:55:18.108982 10690 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:18.109997 10068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" is_pre_election: true
I20251025 01:55:18.111871 10690 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:18.114140 9974 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:18.114384 9974 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:18.114799 10690 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:55:18.122650 10643 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
W20251025 01:55:18.123978 10643 heartbeater.cc:646] Failed to heartbeat to 127.9.125.188:36641 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:18.353391 10694 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 286b4c3025e24a7c8107763fa40d3ab1)
I20251025 01:55:18.353538 10694 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:18.354137 10694 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:18.354612 10000 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "76559300dae44c0d8a17b583a8875b3f" is_pre_election: true
I20251025 01:55:18.354840 10000 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ceaf0e1ce0f4c23b39a5bf4736de5ce in term 1.
I20251025 01:55:18.355398 10046 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f; no voters:
I20251025 01:55:18.355683 10694 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 01:55:18.355811 10694 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader 286b4c3025e24a7c8107763fa40d3ab1)
I20251025 01:55:18.355964 10694 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 1 FOLLOWER]: Advancing to term 2
W20251025 01:55:18.356189 10044 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:18.358769 10694 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:18.359181 10694 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 2 election: Requested vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:18.359521 10000 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "76559300dae44c0d8a17b583a8875b3f"
I20251025 01:55:18.359680 10000 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 1 FOLLOWER]: Advancing to term 2
W20251025 01:55:18.360178 10044 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:18.361517 10000 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ceaf0e1ce0f4c23b39a5bf4736de5ce in term 2.
I20251025 01:55:18.361884 10046 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f; no voters: 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:18.362144 10694 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 2 FOLLOWER]: Leader election won for term 2
I20251025 01:55:18.362475 10694 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 2 LEADER]: Becoming Leader. State: Replica: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, State: Running, Role: LEADER
I20251025 01:55:18.362862 10694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:18.364347 10697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce. Latest consensus state: current_term: 2 leader_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:18.364593 10697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:18.365253 10699 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 01:55:18.366852 10699 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 01:55:18.367590 10699 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:18.367652 10699 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 01:55:18.368611 10699 catalog_manager.cc:1514] Loading token signing keys...
I20251025 01:55:18.369278 10699 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Loaded TSK: 0
I20251025 01:55:18.370078 10699 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 01:55:18.370827 10699 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251025 01:55:18.371491 10699 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251025 01:55:18.380036 10060 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_ebad90bab99347a39c329ef128d4612e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:18.380092 10059 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_083845c9d31747a68c31d8a1804a12aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:18.382263 10051 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_5a6f9a84749b447180f8e258046f31ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:18.382230 10055 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_aaaf9fef59a24b46a5777425aa3d1c5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:18.380862 10056 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_437718d96b1e4760a51380e0f777933e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:18.381894 10052 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_19b5f67b2cfa4e62bc0d061b1c2c8ecf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:18.380765 10054 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_9e3c9d4bfaa242ce9ed8d33fc144f8be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:18.380452 10057 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_39f4a74b6880456585fc71ca55bff11b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:18.380329 10058 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_a2d327beb950460cbf080c4c44d89943"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:18.382019 10053 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_44e4ccc4f2de47198a0599e2577108cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:18.394235 10000 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 2 FOLLOWER]: Refusing update from remote peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20251025 01:55:18.394811 10697 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20251025 01:55:18.395015 10044 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce -> Peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Couldn't send request to peer 286b4c3025e24a7c8107763fa40d3ab1. Status: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 01:55:18.396908 10690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce. Latest consensus state: current_term: 2 leader_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:18.397050 10690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:18.398484 10694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:18.398928 10694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:18.398979 10690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:18.399154 10690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: FOLLOWER
W20251025 01:55:18.400292 10060 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ebad90bab99347a39c329ef128d4612e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:18.404629 10051 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5a6f9a84749b447180f8e258046f31ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:18.404654 10056 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_437718d96b1e4760a51380e0f777933e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:18.404911 10054 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9e3c9d4bfaa242ce9ed8d33fc144f8be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:18.404646 10057 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_39f4a74b6880456585fc71ca55bff11b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:18.405860 10058 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a2d327beb950460cbf080c4c44d89943 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:18.406327 10059 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_083845c9d31747a68c31d8a1804a12aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:18.407016 10055 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_aaaf9fef59a24b46a5777425aa3d1c5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:18.407573 10052 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:18.408036 10053 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_44e4ccc4f2de47198a0599e2577108cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 01:55:18.446075 10060 catalog_manager.cc:2174] created HMS catalog entry for table [id=69e063c1518a4234a260583bd0ba09c5]
I20251025 01:55:18.457650 10060 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_ebad90bab99347a39c329ef128d4612e" } new_table_name: "default.Table_401d389205a14ac191081b5d0df151e6" modify_external_catalogs: true
I20251025 01:55:18.480360 10051 catalog_manager.cc:2174] created HMS catalog entry for table [id=b59ac96bfe3147de99b6697bcaaed652]
I20251025 01:55:18.487529 10051 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_5a6f9a84749b447180f8e258046f31ca" } new_table_name: "default.Table_c18eb2691b2a4c5daf9746ee47a5e451" modify_external_catalogs: true
I20251025 01:55:18.513299 10578 tablet_service.cc:1505] Processing CreateTablet for tablet 8ace5514215445fd9725e4545bd3e1b2 (DEFAULT_TABLE table=default.table_ebad90bab99347a39c329ef128d4612e [id=69e063c1518a4234a260583bd0ba09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.514686 10578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ace5514215445fd9725e4545bd3e1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.515013 10577 tablet_service.cc:1505] Processing CreateTablet for tablet 9c91a696803149fab067caf68ec6c7bf (DEFAULT_TABLE table=default.table_5a6f9a84749b447180f8e258046f31ca [id=b59ac96bfe3147de99b6697bcaaed652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.516078 10577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c91a696803149fab067caf68ec6c7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.520773 10441 tablet_service.cc:1505] Processing CreateTablet for tablet 9c91a696803149fab067caf68ec6c7bf (DEFAULT_TABLE table=default.table_5a6f9a84749b447180f8e258046f31ca [id=b59ac96bfe3147de99b6697bcaaed652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.521579 10442 tablet_service.cc:1505] Processing CreateTablet for tablet 8ace5514215445fd9725e4545bd3e1b2 (DEFAULT_TABLE table=default.table_ebad90bab99347a39c329ef128d4612e [id=69e063c1518a4234a260583bd0ba09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.522219 10441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c91a696803149fab067caf68ec6c7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.522611 10442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ace5514215445fd9725e4545bd3e1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.526244 10301 tablet_service.cc:1505] Processing CreateTablet for tablet 8ace5514215445fd9725e4545bd3e1b2 (DEFAULT_TABLE table=default.table_ebad90bab99347a39c329ef128d4612e [id=69e063c1518a4234a260583bd0ba09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.527014 10300 tablet_service.cc:1505] Processing CreateTablet for tablet 9c91a696803149fab067caf68ec6c7bf (DEFAULT_TABLE table=default.table_5a6f9a84749b447180f8e258046f31ca [id=b59ac96bfe3147de99b6697bcaaed652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.530882 10300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c91a696803149fab067caf68ec6c7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.539566 10301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ace5514215445fd9725e4545bd3e1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.540766 10056 catalog_manager.cc:2174] created HMS catalog entry for table [id=3df0b36649ab414c9c552abbd401d14c]
I20251025 01:55:18.548444 10056 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_437718d96b1e4760a51380e0f777933e" } new_table_name: "default.Table_328c63794848430fb0f74a446d210d94" modify_external_catalogs: true
I20251025 01:55:18.557914 10299 tablet_service.cc:1505] Processing CreateTablet for tablet 899a5108d130421389971072c5802b03 (DEFAULT_TABLE table=default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.558532 10299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899a5108d130421389971072c5802b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.563658 10440 tablet_service.cc:1505] Processing CreateTablet for tablet 899a5108d130421389971072c5802b03 (DEFAULT_TABLE table=default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.564167 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899a5108d130421389971072c5802b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.568382 10576 tablet_service.cc:1505] Processing CreateTablet for tablet 899a5108d130421389971072c5802b03 (DEFAULT_TABLE table=default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.568974 10576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899a5108d130421389971072c5802b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.573359 10714 tablet_bootstrap.cc:492] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:18.574915 10054 catalog_manager.cc:2174] created HMS catalog entry for table [id=91fea98dbb4548a2babcdf286454d811]
I20251025 01:55:18.576520 10713 tablet_bootstrap.cc:492] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:18.577311 10712 tablet_bootstrap.cc:492] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:18.578018 10714 tablet_bootstrap.cc:654] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.579433 10714 log.cc:826] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:18.580745 10712 tablet_bootstrap.cc:654] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.582154 10712 log.cc:826] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:18.582201 10054 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_9e3c9d4bfaa242ce9ed8d33fc144f8be" } new_table_name: "default.Table_8553cdea51804bfa861b6c10537b99c3" modify_external_catalogs: true
I20251025 01:55:18.588411 10712 tablet_bootstrap.cc:492] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:18.589529 10712 ts_tablet_manager.cc:1403] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20251025 01:55:18.590050 10301 tablet_service.cc:1505] Processing CreateTablet for tablet a8bb9f9752b641499955e678b2ccc943 (DEFAULT_TABLE table=default.table_9e3c9d4bfaa242ce9ed8d33fc144f8be [id=91fea98dbb4548a2babcdf286454d811]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.590178 10440 tablet_service.cc:1505] Processing CreateTablet for tablet a8bb9f9752b641499955e678b2ccc943 (DEFAULT_TABLE table=default.table_9e3c9d4bfaa242ce9ed8d33fc144f8be [id=91fea98dbb4548a2babcdf286454d811]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.590420 10301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8bb9f9752b641499955e678b2ccc943. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.590489 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8bb9f9752b641499955e678b2ccc943. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.593371 10577 tablet_service.cc:1505] Processing CreateTablet for tablet a8bb9f9752b641499955e678b2ccc943 (DEFAULT_TABLE table=default.table_9e3c9d4bfaa242ce9ed8d33fc144f8be [id=91fea98dbb4548a2babcdf286454d811]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.593758 10577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8bb9f9752b641499955e678b2ccc943. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.599839 10714 tablet_bootstrap.cc:492] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:18.600152 10714 ts_tablet_manager.cc:1403] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.000s
I20251025 01:55:18.602416 10712 raft_consensus.cc:359] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.602873 10712 raft_consensus.cc:385] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.603008 10712 raft_consensus.cc:740] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.603816 10712 consensus_queue.cc:260] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.604807 10713 tablet_bootstrap.cc:654] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.605991 10713 log.cc:826] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:18.607265 10714 raft_consensus.cc:359] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:18.607735 10714 raft_consensus.cc:385] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.607873 10714 raft_consensus.cc:740] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.608527 10713 tablet_bootstrap.cc:492] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:18.608660 10714 consensus_queue.cc:260] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:18.608825 10713 ts_tablet_manager.cc:1403] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.000s
I20251025 01:55:18.609999 10712 ts_tablet_manager.cc:1434] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.020s user 0.009s sys 0.002s
I20251025 01:55:18.612104 10712 tablet_bootstrap.cc:492] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:18.613941 10712 tablet_bootstrap.cc:654] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.618475 10713 raft_consensus.cc:359] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:18.618840 10714 ts_tablet_manager.cc:1434] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.019s user 0.010s sys 0.000s
I20251025 01:55:18.618877 10713 raft_consensus.cc:385] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.619030 10713 raft_consensus.cc:740] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.619721 10713 consensus_queue.cc:260] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:18.622606 10714 tablet_bootstrap.cc:492] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:18.624716 10714 tablet_bootstrap.cc:654] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.628376 10712 tablet_bootstrap.cc:492] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:18.628520 10712 ts_tablet_manager.cc:1403] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20251025 01:55:18.628814 10713 ts_tablet_manager.cc:1434] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.020s user 0.011s sys 0.005s
I20251025 01:55:18.628830 10714 tablet_bootstrap.cc:492] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:18.628948 10714 ts_tablet_manager.cc:1403] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251025 01:55:18.629376 10712 raft_consensus.cc:359] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:18.629534 10712 raft_consensus.cc:385] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.629608 10712 raft_consensus.cc:740] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.629667 10714 raft_consensus.cc:359] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.629947 10714 raft_consensus.cc:385] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.629966 10712 consensus_queue.cc:260] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:18.630100 10714 raft_consensus.cc:740] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.630426 10714 consensus_queue.cc:260] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.632496 10712 ts_tablet_manager.cc:1434] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 01:55:18.632918 10714 ts_tablet_manager.cc:1434] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 01:55:18.633194 10714 tablet_bootstrap.cc:492] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:18.634644 10714 tablet_bootstrap.cc:654] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.634833 10712 tablet_bootstrap.cc:492] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:18.635869 10713 tablet_bootstrap.cc:492] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:18.636361 10712 tablet_bootstrap.cc:654] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.637508 10713 tablet_bootstrap.cc:654] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.642872 10713 tablet_bootstrap.cc:492] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:18.642994 10713 ts_tablet_manager.cc:1403] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20251025 01:55:18.643244 10714 tablet_bootstrap.cc:492] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:18.643396 10714 ts_tablet_manager.cc:1403] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20251025 01:55:18.643932 10713 raft_consensus.cc:359] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.644088 10713 raft_consensus.cc:385] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.644172 10713 raft_consensus.cc:740] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.644369 10713 consensus_queue.cc:260] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.644471 10712 tablet_bootstrap.cc:492] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:18.644613 10712 ts_tablet_manager.cc:1403] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20251025 01:55:18.644833 10713 ts_tablet_manager.cc:1434] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 01:55:18.645087 10713 tablet_bootstrap.cc:492] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:18.645256 10712 raft_consensus.cc:359] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.645413 10712 raft_consensus.cc:385] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.645496 10712 raft_consensus.cc:740] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.645879 10714 raft_consensus.cc:359] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.646085 10714 raft_consensus.cc:385] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.646162 10714 raft_consensus.cc:740] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.646409 10712 consensus_queue.cc:260] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.646559 10713 tablet_bootstrap.cc:654] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.646910 10712 ts_tablet_manager.cc:1434] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 01:55:18.647186 10712 tablet_bootstrap.cc:492] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:18.648478 10714 consensus_queue.cc:260] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.648638 10712 tablet_bootstrap.cc:654] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.648967 10714 ts_tablet_manager.cc:1434] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251025 01:55:18.649281 10714 tablet_bootstrap.cc:492] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:18.650760 10714 tablet_bootstrap.cc:654] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.653726 10057 catalog_manager.cc:2174] created HMS catalog entry for table [id=57909f703ac64c74b92f656b0b67781f]
W20251025 01:55:18.665761 10510 tablet.cc:2378] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 01:55:18.667506 10645 heartbeater.cc:499] Master 127.9.125.189:45553 was elected leader, sending a full tablet report...
I20251025 01:55:18.682370 10440 tablet_service.cc:1505] Processing CreateTablet for tablet 7772240769e549ff8be322ce7ebc7ce4 (DEFAULT_TABLE table=default.table_39f4a74b6880456585fc71ca55bff11b [id=57909f703ac64c74b92f656b0b67781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.682929 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7772240769e549ff8be322ce7ebc7ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.687319 10577 tablet_service.cc:1505] Processing CreateTablet for tablet 7772240769e549ff8be322ce7ebc7ce4 (DEFAULT_TABLE table=default.table_39f4a74b6880456585fc71ca55bff11b [id=57909f703ac64c74b92f656b0b67781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.687711 10577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7772240769e549ff8be322ce7ebc7ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.692087 10301 tablet_service.cc:1505] Processing CreateTablet for tablet 7772240769e549ff8be322ce7ebc7ce4 (DEFAULT_TABLE table=default.table_39f4a74b6880456585fc71ca55bff11b [id=57909f703ac64c74b92f656b0b67781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.692444 10301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7772240769e549ff8be322ce7ebc7ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.694515 10509 heartbeater.cc:499] Master 127.9.125.189:45553 was elected leader, sending a full tablet report...
I20251025 01:55:18.696390 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.188:36641
--webserver_interface=127.9.125.188
--webserver_port=33371
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251025 01:55:18.703751 10371 heartbeater.cc:499] Master 127.9.125.189:45553 was elected leader, sending a full tablet report...
I20251025 01:55:18.718194 10057 master_service.cc:954] ReplaceTablet: received request to replace tablet 8ace5514215445fd9725e4545bd3e1b2 from {username='slave'} at 127.0.0.1:58658
I20251025 01:55:18.720659 10713 tablet_bootstrap.cc:492] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:18.720969 10713 ts_tablet_manager.cc:1403] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.005s
I20251025 01:55:18.722005 10713 raft_consensus.cc:359] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.722360 10713 raft_consensus.cc:385] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.722499 10713 raft_consensus.cc:740] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.722945 10713 consensus_queue.cc:260] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.725570 10713 ts_tablet_manager.cc:1434] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251025 01:55:18.728394 10713 tablet_bootstrap.cc:492] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:18.730046 10713 tablet_bootstrap.cc:654] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
W20251025 01:55:18.732556 10372 tablet.cc:2378] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 01:55:18.738490 10057 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Sending DeleteTablet for 3 replicas of tablet 8ace5514215445fd9725e4545bd3e1b2
I20251025 01:55:18.739627 10577 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ace5514215445fd9725e4545bd3e1b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f9bd9812cc74f49b4028e89d2b5f0d7) from {username='slave'} at 127.0.0.1:43648
I20251025 01:55:18.740406 10301 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ace5514215445fd9725e4545bd3e1b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f9bd9812cc74f49b4028e89d2b5f0d7) from {username='slave'} at 127.0.0.1:49346
I20251025 01:55:18.740603 10440 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ace5514215445fd9725e4545bd3e1b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f9bd9812cc74f49b4028e89d2b5f0d7) from {username='slave'} at 127.0.0.1:58364
I20251025 01:55:18.745303 10713 tablet_bootstrap.cc:492] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:18.745469 10713 ts_tablet_manager.cc:1403] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20251025 01:55:18.746380 10713 raft_consensus.cc:359] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.746572 10713 raft_consensus.cc:385] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.746671 10713 raft_consensus.cc:740] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.746905 10713 consensus_queue.cc:260] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.747438 10713 ts_tablet_manager.cc:1434] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 01:55:18.747680 10713 tablet_bootstrap.cc:492] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:18.758518 10058 catalog_manager.cc:2174] created HMS catalog entry for table [id=7477ea6c70f84bc2897964f18294d49f]
I20251025 01:55:18.763195 10713 tablet_bootstrap.cc:654] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.765311 10441 tablet_service.cc:1505] Processing CreateTablet for tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 (DEFAULT_TABLE table=default.table_ebad90bab99347a39c329ef128d4612e [id=69e063c1518a4234a260583bd0ba09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.765762 10441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f9bd9812cc74f49b4028e89d2b5f0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.765856 10732 tablet_replica.cc:333] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:18.766337 10732 raft_consensus.cc:2243] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:18.766726 10732 raft_consensus.cc:2272] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:18.768424 10578 tablet_service.cc:1505] Processing CreateTablet for tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 (DEFAULT_TABLE table=default.table_ebad90bab99347a39c329ef128d4612e [id=69e063c1518a4234a260583bd0ba09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.768792 10578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f9bd9812cc74f49b4028e89d2b5f0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.774883 10301 tablet_service.cc:1505] Processing CreateTablet for tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 (DEFAULT_TABLE table=default.table_ebad90bab99347a39c329ef128d4612e [id=69e063c1518a4234a260583bd0ba09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.775655 10301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f9bd9812cc74f49b4028e89d2b5f0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.776118 10731 tablet_replica.cc:333] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:18.776275 10058 master_service.cc:954] ReplaceTablet: received request to replace tablet 899a5108d130421389971072c5802b03 from {username='slave'} at 127.0.0.1:58658
I20251025 01:55:18.778658 10717 raft_consensus.cc:493] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:18.778894 10717 raft_consensus.cc:515] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:18.780743 10717 leader_election.cc:290] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:18.780822 10731 raft_consensus.cc:2243] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:18.781185 10731 raft_consensus.cc:2272] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:18.786264 10731 ts_tablet_manager.cc:1916] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:18.787442 10732 ts_tablet_manager.cc:1916] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:18.789086 10719 raft_consensus.cc:493] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:18.792058 10719 raft_consensus.cc:515] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.793926 10719 leader_election.cc:290] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:18.802217 10058 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Sending DeleteTablet for 3 replicas of tablet 899a5108d130421389971072c5802b03
I20251025 01:55:18.802716 10058 catalog_manager.cc:6719] ReplaceTablet: tablet 899a5108d130421389971072c5802b03 deleted and replaced by tablet df08f7a1b9ca4b8e8328d9295bf52576
I20251025 01:55:18.804654 10058 master_service.cc:954] ReplaceTablet: received request to replace tablet 899a5108d130421389971072c5802b03 from {username='slave'} at 127.0.0.1:58658
I20251025 01:55:18.805130 10058 master_service.cc:954] ReplaceTablet: received request to replace tablet 9c91a696803149fab067caf68ec6c7bf from {username='slave'} at 127.0.0.1:58658
I20251025 01:55:18.806972 10299 tablet_service.cc:1552] Processing DeleteTablet for tablet 899a5108d130421389971072c5802b03 with delete_type TABLET_DATA_DELETED (replaced by tablet df08f7a1b9ca4b8e8328d9295bf52576) from {username='slave'} at 127.0.0.1:49346
I20251025 01:55:18.823872 10058 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Sending DeleteTablet for 3 replicas of tablet 9c91a696803149fab067caf68ec6c7bf
I20251025 01:55:18.826910 10732 ts_tablet_manager.cc:1929] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:18.829175 10717 raft_consensus.cc:493] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:18.829348 10717 raft_consensus.cc:515] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.829989 10717 leader_election.cc:290] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:18.832289 10578 tablet_service.cc:1505] Processing CreateTablet for tablet bab272a511e14adab00dd094ff1ee9e5 (DEFAULT_TABLE table=default.table_a2d327beb950460cbf080c4c44d89943 [id=7477ea6c70f84bc2897964f18294d49f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.832746 10578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bab272a511e14adab00dd094ff1ee9e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.835736 10577 tablet_service.cc:1552] Processing DeleteTablet for tablet 899a5108d130421389971072c5802b03 with delete_type TABLET_DATA_DELETED (replaced by tablet df08f7a1b9ca4b8e8328d9295bf52576) from {username='slave'} at 127.0.0.1:43648
I20251025 01:55:18.839363 10058 catalog_manager.cc:6719] ReplaceTablet: tablet 9c91a696803149fab067caf68ec6c7bf deleted and replaced by tablet f13a1ad12e2f4f2c95103cf8d76d6da6
I20251025 01:55:18.840274 10058 master_service.cc:954] ReplaceTablet: received request to replace tablet 9c91a696803149fab067caf68ec6c7bf from {username='slave'} at 127.0.0.1:58658
I20251025 01:55:18.841981 10057 catalog_manager.cc:6719] ReplaceTablet: tablet 8ace5514215445fd9725e4545bd3e1b2 deleted and replaced by tablet 9f9bd9812cc74f49b4028e89d2b5f0d7
I20251025 01:55:18.842453 10057 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_cf46809300e645fca4b99a2da934b149"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:18.844101 10058 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_39f4a74b6880456585fc71ca55bff11b" } new_table_name: "default.Table_827ca3c8c36543b29dac05ab520c552a" modify_external_catalogs: true
I20251025 01:55:18.845016 10733 tablet_replica.cc:333] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:18.845424 10577 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c91a696803149fab067caf68ec6c7bf with delete_type TABLET_DATA_DELETED (replaced by tablet f13a1ad12e2f4f2c95103cf8d76d6da6) from {username='slave'} at 127.0.0.1:43648
I20251025 01:55:18.845585 10733 raft_consensus.cc:2243] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:18.846055 10733 raft_consensus.cc:2272] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:18.847692 10059 catalog_manager.cc:2174] created HMS catalog entry for table [id=4ba51d95c7ed483987cb002716f0c956]
I20251025 01:55:18.847810 10733 ts_tablet_manager.cc:1916] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:18.848367 10712 tablet_bootstrap.cc:492] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:18.851729 10712 ts_tablet_manager.cc:1403] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.205s user 0.019s sys 0.051s
I20251025 01:55:18.852429 10714 tablet_bootstrap.cc:492] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:18.852658 10714 ts_tablet_manager.cc:1403] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.203s user 0.044s sys 0.064s
I20251025 01:55:18.852643 10712 raft_consensus.cc:359] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.853116 10712 raft_consensus.cc:385] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.853281 10712 raft_consensus.cc:740] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.853596 10712 consensus_queue.cc:260] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.853583 10714 raft_consensus.cc:359] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.853757 10059 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_986ff2b14daa4beebda43706d47dc98b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:18.853936 10714 raft_consensus.cc:385] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.854053 10714 raft_consensus.cc:740] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.854291 10714 consensus_queue.cc:260] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.854748 10714 ts_tablet_manager.cc:1434] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 01:55:18.855084 10714 tablet_bootstrap.cc:492] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:18.855635 10712 ts_tablet_manager.cc:1434] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 01:55:18.856123 10712 tablet_bootstrap.cc:492] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:18.856760 10714 tablet_bootstrap.cc:654] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.859352 10712 tablet_bootstrap.cc:654] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.827385 10299 tablet_service.cc:1505] Processing CreateTablet for tablet bab272a511e14adab00dd094ff1ee9e5 (DEFAULT_TABLE table=default.table_a2d327beb950460cbf080c4c44d89943 [id=7477ea6c70f84bc2897964f18294d49f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.832353 10721 raft_consensus.cc:493] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:18.860451 10721 raft_consensus.cc:515] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.862624 10721 leader_election.cc:290] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:18.845932 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c91a696803149fab067caf68ec6c7bf" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:18.863065 10321 raft_consensus.cc:2468] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:18.851276 10300 tablet_service.cc:1505] Processing CreateTablet for tablet df08f7a1b9ca4b8e8328d9295bf52576 (DEFAULT_TABLE table=default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.864197 10299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bab272a511e14adab00dd094ff1ee9e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.868363 10300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df08f7a1b9ca4b8e8328d9295bf52576. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.874825 10298 tablet_service.cc:1505] Processing CreateTablet for tablet f13a1ad12e2f4f2c95103cf8d76d6da6 (DEFAULT_TABLE table=default.table_5a6f9a84749b447180f8e258046f31ca [id=b59ac96bfe3147de99b6697bcaaed652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.875427 10298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f13a1ad12e2f4f2c95103cf8d76d6da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.852994 10301 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c91a696803149fab067caf68ec6c7bf with delete_type TABLET_DATA_DELETED (replaced by tablet f13a1ad12e2f4f2c95103cf8d76d6da6) from {username='slave'} at 127.0.0.1:49346
I20251025 01:55:18.883634 10731 ts_tablet_manager.cc:1929] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:18.883911 10731 log.cc:1199] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/8ace5514215445fd9725e4545bd3e1b2
I20251025 01:55:18.886281 10731 ts_tablet_manager.cc:1950] T 8ace5514215445fd9725e4545bd3e1b2 P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:18.888355 10731 tablet_replica.cc:333] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:18.888489 10532 leader_election.cc:304] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:18.888888 10719 raft_consensus.cc:2804] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:18.889019 10719 raft_consensus.cc:493] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:18.889106 10719 raft_consensus.cc:3060] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:18.891602 10732 log.cc:1199] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/8ace5514215445fd9725e4545bd3e1b2
I20251025 01:55:18.892164 10732 ts_tablet_manager.cc:1950] T 8ace5514215445fd9725e4545bd3e1b2 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:18.890182 10577 tablet_service.cc:1505] Processing CreateTablet for tablet df08f7a1b9ca4b8e8328d9295bf52576 (DEFAULT_TABLE table=default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.892122 10719 raft_consensus.cc:515] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.892554 10577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df08f7a1b9ca4b8e8328d9295bf52576. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.892788 10719 leader_election.cc:290] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Requested vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:18.893988 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c91a696803149fab067caf68ec6c7bf" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
I20251025 01:55:18.893996 10045 catalog_manager.cc:4985] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): tablet 8ace5514215445fd9725e4545bd3e1b2 (table default.table_ebad90bab99347a39c329ef128d4612e [id=69e063c1518a4234a260583bd0ba09c5]) successfully deleted
I20251025 01:55:18.894151 10321 raft_consensus.cc:3060] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:18.894639 10045 catalog_manager.cc:4985] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): tablet 8ace5514215445fd9725e4545bd3e1b2 (table default.table_ebad90bab99347a39c329ef128d4612e [id=69e063c1518a4234a260583bd0ba09c5]) successfully deleted
W20251025 01:55:18.883730 10646 tablet.cc:2378] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 01:55:18.896189 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c91a696803149fab067caf68ec6c7bf" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:18.896836 10598 raft_consensus.cc:2393] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3e4f9f2253e416e8b31ea86ead23a3b in current term 1: Already voted for candidate 90109d2fdb464942b20891b5856f8ac2 in this term.
I20251025 01:55:18.897320 10321 raft_consensus.cc:2468] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:18.897707 10532 leader_election.cc:304] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:18.898629 10719 raft_consensus.cc:2804] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:18.865196 10733 ts_tablet_manager.cc:1929] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:18.901183 10733 log.cc:1199] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/8ace5514215445fd9725e4545bd3e1b2
I20251025 01:55:18.901672 10733 ts_tablet_manager.cc:1950] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:18.900383 10462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c91a696803149fab067caf68ec6c7bf" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:18.902871 10462 raft_consensus.cc:2468] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:18.900494 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c91a696803149fab067caf68ec6c7bf" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:18.903334 10461 raft_consensus.cc:3060] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:18.903368 10441 tablet_service.cc:1505] Processing CreateTablet for tablet bab272a511e14adab00dd094ff1ee9e5 (DEFAULT_TABLE table=default.table_a2d327beb950460cbf080c4c44d89943 [id=7477ea6c70f84bc2897964f18294d49f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.903788 10441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bab272a511e14adab00dd094ff1ee9e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.906494 10732 tablet_replica.cc:333] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:18.906690 10732 raft_consensus.cc:2243] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:18.906659 10461 raft_consensus.cc:2468] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:18.900831 10440 tablet_service.cc:1552] Processing DeleteTablet for tablet 899a5108d130421389971072c5802b03 with delete_type TABLET_DATA_DELETED (replaced by tablet df08f7a1b9ca4b8e8328d9295bf52576) from {username='slave'} at 127.0.0.1:58364
I20251025 01:55:18.907281 10732 raft_consensus.cc:2272] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:18.907334 10733 tablet_replica.cc:333] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:18.907502 10733 raft_consensus.cc:2243] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:18.907662 10733 raft_consensus.cc:2272] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:18.908535 10733 ts_tablet_manager.cc:1916] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:18.909741 10732 ts_tablet_manager.cc:1916] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:18.911026 10440 tablet_service.cc:1505] Processing CreateTablet for tablet df08f7a1b9ca4b8e8328d9295bf52576 (DEFAULT_TABLE table=default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.911450 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df08f7a1b9ca4b8e8328d9295bf52576. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.914988 10733 ts_tablet_manager.cc:1929] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:18.916411 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c91a696803149fab067caf68ec6c7bf" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:18.917518 10461 raft_consensus.cc:2393] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3e4f9f2253e416e8b31ea86ead23a3b in current term 1: Already voted for candidate 90109d2fdb464942b20891b5856f8ac2 in this term.
I20251025 01:55:18.918366 10254 leader_election.cc:304] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:55:18.918893 10721 raft_consensus.cc:2749] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:55:18.921756 10732 ts_tablet_manager.cc:1929] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:18.922732 10732 log.cc:1199] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/899a5108d130421389971072c5802b03
I20251025 01:55:18.923389 10732 ts_tablet_manager.cc:1950] T 899a5108d130421389971072c5802b03 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:18.924839 10045 catalog_manager.cc:4985] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): tablet 899a5108d130421389971072c5802b03 (table default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c]) successfully deleted
I20251025 01:55:18.925411 10731 raft_consensus.cc:2243] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:18.925602 10731 raft_consensus.cc:2272] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:18.926308 10731 ts_tablet_manager.cc:1916] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:18.930403 10578 tablet_service.cc:1505] Processing CreateTablet for tablet f13a1ad12e2f4f2c95103cf8d76d6da6 (DEFAULT_TABLE table=default.table_5a6f9a84749b447180f8e258046f31ca [id=b59ac96bfe3147de99b6697bcaaed652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.931367 10578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f13a1ad12e2f4f2c95103cf8d76d6da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.934692 10731 ts_tablet_manager.cc:1929] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:18.934840 10731 log.cc:1199] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/899a5108d130421389971072c5802b03
I20251025 01:55:18.935148 10733 log.cc:1199] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/899a5108d130421389971072c5802b03
I20251025 01:55:18.935622 10044 catalog_manager.cc:4985] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): tablet 8ace5514215445fd9725e4545bd3e1b2 (table default.table_ebad90bab99347a39c329ef128d4612e [id=69e063c1518a4234a260583bd0ba09c5]) successfully deleted
I20251025 01:55:18.935662 10733 ts_tablet_manager.cc:1950] T 899a5108d130421389971072c5802b03 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:18.937009 10298 tablet_service.cc:1505] Processing CreateTablet for tablet 47b9e247969945439741dea4298b99ab (DEFAULT_TABLE table=default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.937539 10298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47b9e247969945439741dea4298b99ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.938460 10712 tablet_bootstrap.cc:492] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:18.938680 10712 ts_tablet_manager.cc:1403] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.083s user 0.005s sys 0.005s
I20251025 01:55:18.938719 10576 tablet_service.cc:1505] Processing CreateTablet for tablet 47b9e247969945439741dea4298b99ab (DEFAULT_TABLE table=default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.939241 10576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47b9e247969945439741dea4298b99ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.939797 10712 raft_consensus.cc:359] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.940099 10712 raft_consensus.cc:385] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.940203 10712 raft_consensus.cc:740] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.940450 10712 consensus_queue.cc:260] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.942560 10712 ts_tablet_manager.cc:1434] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 01:55:18.943642 10714 tablet_bootstrap.cc:492] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:18.943797 10714 ts_tablet_manager.cc:1403] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.001s
I20251025 01:55:18.944495 10712 tablet_bootstrap.cc:492] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:18.944537 10714 raft_consensus.cc:359] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.944734 10714 raft_consensus.cc:385] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.944859 10714 raft_consensus.cc:740] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.945147 10732 tablet_replica.cc:333] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:18.945112 10714 consensus_queue.cc:260] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.946656 10712 tablet_bootstrap.cc:654] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.946884 10732 raft_consensus.cc:2243] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:18.947158 10732 raft_consensus.cc:2272] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:18.947768 10714 ts_tablet_manager.cc:1434] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 01:55:18.948078 10732 ts_tablet_manager.cc:1916] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:18.948415 10714 tablet_bootstrap.cc:492] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:18.949373 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c91a696803149fab067caf68ec6c7bf" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:18.950088 10714 tablet_bootstrap.cc:654] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.951009 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c91a696803149fab067caf68ec6c7bf" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:18.951176 10731 ts_tablet_manager.cc:1950] T 899a5108d130421389971072c5802b03 P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:18.951684 10441 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c91a696803149fab067caf68ec6c7bf with delete_type TABLET_DATA_DELETED (replaced by tablet f13a1ad12e2f4f2c95103cf8d76d6da6) from {username='slave'} at 127.0.0.1:58364
I20251025 01:55:18.951762 10044 catalog_manager.cc:4985] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): tablet 899a5108d130421389971072c5802b03 (table default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c]) successfully deleted
I20251025 01:55:18.952653 10731 tablet_replica.cc:333] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:18.952796 10442 tablet_service.cc:1505] Processing CreateTablet for tablet 47b9e247969945439741dea4298b99ab (DEFAULT_TABLE table=default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.953199 10045 catalog_manager.cc:4985] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): tablet 899a5108d130421389971072c5802b03 (table default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c]) successfully deleted
I20251025 01:55:18.953528 10442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47b9e247969945439741dea4298b99ab. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 01:55:18.953892 10396 leader_election.cc:343] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 8ace5514215445fd9725e4545bd3e1b2
I20251025 01:55:18.957520 10732 ts_tablet_manager.cc:1929] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:18.947530 10440 tablet_service.cc:1505] Processing CreateTablet for tablet f13a1ad12e2f4f2c95103cf8d76d6da6 (DEFAULT_TABLE table=default.table_5a6f9a84749b447180f8e258046f31ca [id=b59ac96bfe3147de99b6697bcaaed652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:18.957693 10732 log.cc:1199] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/9c91a696803149fab067caf68ec6c7bf
I20251025 01:55:18.957182 10712 tablet_bootstrap.cc:492] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:18.957975 10712 ts_tablet_manager.cc:1403] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20251025 01:55:18.958073 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f13a1ad12e2f4f2c95103cf8d76d6da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:18.958842 10712 raft_consensus.cc:359] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.959123 10712 raft_consensus.cc:385] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.959300 10712 raft_consensus.cc:740] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.959585 10712 consensus_queue.cc:260] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.960783 10733 tablet_replica.cc:333] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:18.961012 10733 raft_consensus.cc:2243] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:18.961185 10713 tablet_bootstrap.cc:492] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:18.961198 10733 raft_consensus.cc:2272] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:18.961319 10713 ts_tablet_manager.cc:1403] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.214s user 0.037s sys 0.017s
I20251025 01:55:18.962002 10713 raft_consensus.cc:359] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.962167 10713 raft_consensus.cc:385] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.962222 10713 raft_consensus.cc:740] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.962469 10713 consensus_queue.cc:260] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.962915 10713 ts_tablet_manager.cc:1434] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 01:55:18.963231 10713 tablet_bootstrap.cc:492] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
W20251025 01:55:18.964217 10396 leader_election.cc:343] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Illegal state: Tablet not RUNNING: STOPPED
W20251025 01:55:18.964402 10396 leader_election.cc:343] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8ace5514215445fd9725e4545bd3e1b2
I20251025 01:55:18.964547 10396 leader_election.cc:304] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c; no voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:55:18.964792 10713 tablet_bootstrap.cc:654] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.965677 10714 tablet_bootstrap.cc:492] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:18.965718 10719 raft_consensus.cc:697] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Running, Role: LEADER
I20251025 01:55:18.965970 10714 ts_tablet_manager.cc:1403] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.003s
I20251025 01:55:18.966321 10719 consensus_queue.cc:237] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.966899 10714 raft_consensus.cc:359] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.967144 10714 raft_consensus.cc:385] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.967321 10714 raft_consensus.cc:740] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.967617 10714 consensus_queue.cc:260] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.968168 10714 ts_tablet_manager.cc:1434] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251025 01:55:18.965807 10396 raft_consensus.cc:2686] T 8ace5514215445fd9725e4545bd3e1b2 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251025 01:55:18.968959 10714 tablet_bootstrap.cc:492] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:18.967096 10733 ts_tablet_manager.cc:1916] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:18.969247 10732 ts_tablet_manager.cc:1950] T 9c91a696803149fab067caf68ec6c7bf P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:18.968926 10712 ts_tablet_manager.cc:1434] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20251025 01:55:18.970708 10714 tablet_bootstrap.cc:654] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.972337 10712 tablet_bootstrap.cc:492] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:18.974766 10712 tablet_bootstrap.cc:654] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.976215 10045 catalog_manager.cc:4985] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): tablet 9c91a696803149fab067caf68ec6c7bf (table default.table_5a6f9a84749b447180f8e258046f31ca [id=b59ac96bfe3147de99b6697bcaaed652]) successfully deleted
I20251025 01:55:18.976140 10733 ts_tablet_manager.cc:1929] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:18.977715 10733 log.cc:1199] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/9c91a696803149fab067caf68ec6c7bf
I20251025 01:55:18.978112 10733 ts_tablet_manager.cc:1950] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:18.978509 10714 tablet_bootstrap.cc:492] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:18.978991 10714 ts_tablet_manager.cc:1403] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20251025 01:55:18.979207 10044 catalog_manager.cc:4985] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): tablet 9c91a696803149fab067caf68ec6c7bf (table default.table_5a6f9a84749b447180f8e258046f31ca [id=b59ac96bfe3147de99b6697bcaaed652]) successfully deleted
I20251025 01:55:18.980168 10713 tablet_bootstrap.cc:492] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:18.980319 10713 ts_tablet_manager.cc:1403] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20251025 01:55:18.980652 10714 raft_consensus.cc:359] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.981123 10714 raft_consensus.cc:385] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.981073 10713 raft_consensus.cc:359] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.981271 10713 raft_consensus.cc:385] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.981374 10713 raft_consensus.cc:740] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.981371 10714 raft_consensus.cc:740] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.981604 10713 consensus_queue.cc:260] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.981765 10714 consensus_queue.cc:260] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.982568 10714 ts_tablet_manager.cc:1434] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 01:55:18.982946 10714 tablet_bootstrap.cc:492] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:18.983932 10713 ts_tablet_manager.cc:1434] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 01:55:18.985415 10714 tablet_bootstrap.cc:654] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.985453 10713 tablet_bootstrap.cc:492] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:18.987141 10713 tablet_bootstrap.cc:654] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.987164 10712 tablet_bootstrap.cc:492] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:18.987447 10712 ts_tablet_manager.cc:1403] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20251025 01:55:18.987545 10731 raft_consensus.cc:2243] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:18.988211 10712 raft_consensus.cc:359] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.988423 10731 pending_rounds.cc:70] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2: Trying to abort 1 pending ops.
I20251025 01:55:18.988565 10712 raft_consensus.cc:385] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.988605 10731 pending_rounds.cc:77] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214519578491273216 op_type: NO_OP noop_request { }
I20251025 01:55:18.988685 10712 raft_consensus.cc:740] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.988807 10731 raft_consensus.cc:2889] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 01:55:18.988935 10731 raft_consensus.cc:2272] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:18.989077 10712 consensus_queue.cc:260] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.990326 10712 ts_tablet_manager.cc:1434] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251025 01:55:18.990674 10712 tablet_bootstrap.cc:492] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:18.992308 10712 tablet_bootstrap.cc:654] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:18.993806 10731 ts_tablet_manager.cc:1916] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:18.994971 10713 tablet_bootstrap.cc:492] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:18.995119 10713 ts_tablet_manager.cc:1403] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20251025 01:55:18.995805 10713 raft_consensus.cc:359] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:18.995994 10713 raft_consensus.cc:385] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:18.996068 10713 raft_consensus.cc:740] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:18.996301 10713 consensus_queue.cc:260] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
W20251025 01:55:18.992640 10396 leader_election.cc:343] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Illegal state: must be running to vote when last-logged opid is not known
I20251025 01:55:18.999781 10396 leader_election.cc:304] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c; no voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:55:19.000020 10713 ts_tablet_manager.cc:1434] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251025 01:55:19.000666 10731 ts_tablet_manager.cc:1929] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:19.000824 10731 log.cc:1199] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/9c91a696803149fab067caf68ec6c7bf
I20251025 01:55:19.001241 10731 ts_tablet_manager.cc:1950] T 9c91a696803149fab067caf68ec6c7bf P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:19.002344 10045 catalog_manager.cc:4985] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): tablet 9c91a696803149fab067caf68ec6c7bf (table default.table_5a6f9a84749b447180f8e258046f31ca [id=b59ac96bfe3147de99b6697bcaaed652]) successfully deleted
W20251025 01:55:19.002656 10396 raft_consensus.cc:2686] T 9c91a696803149fab067caf68ec6c7bf P 1f6e00cb735e4cb2bd1221cdcbc3c62c: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251025 01:55:19.002943 10713 tablet_bootstrap.cc:492] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:19.004812 10713 tablet_bootstrap.cc:654] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.007766 10714 tablet_bootstrap.cc:492] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:19.007910 10714 ts_tablet_manager.cc:1403] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.006s
I20251025 01:55:19.008611 10714 raft_consensus.cc:359] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.008761 10714 raft_consensus.cc:385] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.008827 10714 raft_consensus.cc:740] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.009060 10714 consensus_queue.cc:260] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.010934 10714 ts_tablet_manager.cc:1434] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251025 01:55:19.011878 10713 tablet_bootstrap.cc:492] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:19.011933 10714 tablet_bootstrap.cc:492] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:19.012032 10713 ts_tablet_manager.cc:1403] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20251025 01:55:19.012713 10713 raft_consensus.cc:359] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.012879 10713 raft_consensus.cc:385] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.012966 10713 raft_consensus.cc:740] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.013206 10713 consensus_queue.cc:260] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.013444 10714 tablet_bootstrap.cc:654] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.013622 10713 ts_tablet_manager.cc:1434] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251025 01:55:19.013865 10713 tablet_bootstrap.cc:492] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:19.014758 10712 tablet_bootstrap.cc:492] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:19.014940 10712 ts_tablet_manager.cc:1403] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.010s
I20251025 01:55:19.015666 10713 tablet_bootstrap.cc:654] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.015760 10712 raft_consensus.cc:359] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.015991 10712 raft_consensus.cc:385] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.022903 10712 raft_consensus.cc:740] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.023383 10712 consensus_queue.cc:260] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.027135 10712 ts_tablet_manager.cc:1434] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251025 01:55:19.030320 10712 tablet_bootstrap.cc:492] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:19.032553 10712 tablet_bootstrap.cc:654] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.044924 10714 tablet_bootstrap.cc:492] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:19.044920 10713 tablet_bootstrap.cc:492] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:19.045085 10713 ts_tablet_manager.cc:1403] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.031s user 0.016s sys 0.012s
I20251025 01:55:19.045084 10714 ts_tablet_manager.cc:1403] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.017s
I20251025 01:55:19.045871 10713 raft_consensus.cc:359] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.045923 10714 raft_consensus.cc:359] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.046113 10713 raft_consensus.cc:385] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.046126 10714 raft_consensus.cc:385] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.046187 10713 raft_consensus.cc:740] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.046221 10714 raft_consensus.cc:740] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.046443 10713 consensus_queue.cc:260] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.046533 10714 consensus_queue.cc:260] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.050168 10712 tablet_bootstrap.cc:492] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:19.050375 10712 ts_tablet_manager.cc:1403] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20251025 01:55:19.051102 10712 raft_consensus.cc:359] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.051414 10712 raft_consensus.cc:385] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.051424 10713 ts_tablet_manager.cc:1434] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251025 01:55:19.051544 10712 raft_consensus.cc:740] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.051826 10712 consensus_queue.cc:260] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.054983 10712 ts_tablet_manager.cc:1434] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251025 01:55:19.055924 10714 ts_tablet_manager.cc:1434] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251025 01:55:19.057031 10714 tablet_bootstrap.cc:492] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:19.057495 10713 tablet_bootstrap.cc:492] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:19.058571 10714 tablet_bootstrap.cc:654] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.059064 10713 tablet_bootstrap.cc:654] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.060662 10712 tablet_bootstrap.cc:492] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:19.062143 10712 tablet_bootstrap.cc:654] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.080260 10713 tablet_bootstrap.cc:492] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:19.081554 10713 ts_tablet_manager.cc:1403] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20251025 01:55:19.082820 10713 raft_consensus.cc:359] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.083109 10713 raft_consensus.cc:385] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.083195 10713 raft_consensus.cc:740] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.083609 10713 consensus_queue.cc:260] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.086606 10713 ts_tablet_manager.cc:1434] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 01:55:19.087971 10714 tablet_bootstrap.cc:492] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:19.088135 10714 ts_tablet_manager.cc:1403] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.008s
I20251025 01:55:19.089025 10714 raft_consensus.cc:359] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.089236 10714 raft_consensus.cc:385] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.089322 10714 raft_consensus.cc:740] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.089556 10714 consensus_queue.cc:260] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.090137 10714 ts_tablet_manager.cc:1434] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 01:55:19.092358 10712 tablet_bootstrap.cc:492] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:19.092562 10712 ts_tablet_manager.cc:1403] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.014s
I20251025 01:55:19.093382 10712 raft_consensus.cc:359] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.095481 10712 raft_consensus.cc:385] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.095602 10712 raft_consensus.cc:740] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.095855 10712 consensus_queue.cc:260] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.096585 10712 ts_tablet_manager.cc:1434] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
01:55:19.143 [HMS - ERROR - pool-6-thread-3] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_aaaf9fef59a24b46a5777425aa3d1c5b already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy26.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20251025 01:55:19.146005 10055 catalog_manager.cc:2170] Already present: failed to create HMS catalog entry for table [id=46353c98fb6647b29d746bcbc828f88b]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_aaaf9fef59a24b46a5777425aa3d1c5b already exists)
I20251025 01:55:19.155455 10055 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_ac99b534589f45e6871ddfa3a1331b16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:19.177513 10719 raft_consensus.cc:493] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:19.177726 10719 raft_consensus.cc:515] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.178349 10719 leader_election.cc:290] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:19.179621 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47b9e247969945439741dea4298b99ab" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:19.179836 10321 raft_consensus.cc:2468] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:19.180147 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47b9e247969945439741dea4298b99ab" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:19.180346 10461 raft_consensus.cc:2468] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:19.180919 10531 leader_election.cc:304] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2; no voters:
I20251025 01:55:19.181635 10719 raft_consensus.cc:2804] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:19.181727 10719 raft_consensus.cc:493] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:19.181826 10719 raft_consensus.cc:3060] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.184791 10719 raft_consensus.cc:515] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.185720 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47b9e247969945439741dea4298b99ab" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
I20251025 01:55:19.185894 10321 raft_consensus.cc:3060] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.188576 10321 raft_consensus.cc:2468] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:19.189000 10532 leader_election.cc:304] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.189270 10740 raft_consensus.cc:2804] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:19.189394 10740 raft_consensus.cc:697] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Running, Role: LEADER
I20251025 01:55:19.189684 10740 consensus_queue.cc:237] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.191689 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47b9e247969945439741dea4298b99ab" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:19.191905 10461 raft_consensus.cc:3060] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.194658 10461 raft_consensus.cc:2468] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:19.195392 10719 leader_election.cc:290] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Requested vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:19.206771 10052 catalog_manager.cc:2174] created HMS catalog entry for table [id=43572de558b840adbf7499428842e812]
I20251025 01:55:19.228010 10298 tablet_service.cc:1505] Processing CreateTablet for tablet dcf81923a06d4750bbec550cff9bd91d (DEFAULT_TABLE table=default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.228277 10440 tablet_service.cc:1505] Processing CreateTablet for tablet dcf81923a06d4750bbec550cff9bd91d (DEFAULT_TABLE table=default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.228550 10298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf81923a06d4750bbec550cff9bd91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:19.228710 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf81923a06d4750bbec550cff9bd91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:19.230758 10577 tablet_service.cc:1505] Processing CreateTablet for tablet dcf81923a06d4750bbec550cff9bd91d (DEFAULT_TABLE table=default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.231212 10577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf81923a06d4750bbec550cff9bd91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:19.233906 10721 raft_consensus.cc:493] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:19.234067 10721 raft_consensus.cc:515] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.234725 10713 tablet_bootstrap.cc:492] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:19.234812 10721 leader_election.cc:290] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:19.236404 10713 tablet_bootstrap.cc:654] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.237602 10714 tablet_bootstrap.cc:492] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:19.237607 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df08f7a1b9ca4b8e8328d9295bf52576" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:19.237823 10461 raft_consensus.cc:2468] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:19.238179 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df08f7a1b9ca4b8e8328d9295bf52576" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:19.238420 10598 raft_consensus.cc:2468] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:19.238883 10255 leader_election.cc:304] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.239315 10714 tablet_bootstrap.cc:654] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.239534 10712 tablet_bootstrap.cc:492] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:19.239531 10721 raft_consensus.cc:2804] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:19.239637 10721 raft_consensus.cc:493] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:19.239720 10721 raft_consensus.cc:3060] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.241182 10712 tablet_bootstrap.cc:654] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.242939 10721 raft_consensus.cc:515] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.243186 10052 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_a2d327beb950460cbf080c4c44d89943" } modify_external_catalogs: true
I20251025 01:55:19.243520 10052 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_a2d327beb950460cbf080c4c44d89943" } modify_external_catalogs: true
I20251025 01:55:19.243899 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df08f7a1b9ca4b8e8328d9295bf52576" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:19.244053 10461 raft_consensus.cc:3060] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.246155 10717 raft_consensus.cc:493] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:19.246294 10717 raft_consensus.cc:515] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.246868 10717 leader_election.cc:290] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:19.247217 10461 raft_consensus.cc:2468] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:19.247581 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f9bd9812cc74f49b4028e89d2b5f0d7" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:19.247808 10321 raft_consensus.cc:2468] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 0.
I20251025 01:55:19.247855 10254 leader_election.cc:304] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.248279 10396 leader_election.cc:304] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.248793 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f9bd9812cc74f49b4028e89d2b5f0d7" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:19.248997 10598 raft_consensus.cc:2468] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 0.
I20251025 01:55:19.249513 10717 raft_consensus.cc:2804] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:19.249620 10717 raft_consensus.cc:493] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:19.249697 10717 raft_consensus.cc:3060] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.252179 10717 raft_consensus.cc:515] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.253037 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f9bd9812cc74f49b4028e89d2b5f0d7" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
I20251025 01:55:19.253221 10321 raft_consensus.cc:3060] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.254024 10721 leader_election.cc:290] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:19.254271 10721 raft_consensus.cc:2804] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:19.254426 10721 raft_consensus.cc:697] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:19.254889 10721 consensus_queue.cc:237] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.255815 10321 raft_consensus.cc:2468] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 1.
I20251025 01:55:19.256222 10396 leader_election.cc:304] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.256608 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df08f7a1b9ca4b8e8328d9295bf52576" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:19.256779 10598 raft_consensus.cc:3060] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.257019 10053 catalog_manager.cc:2174] created HMS catalog entry for table [id=58afab4104e84c868eeccda411b8452b]
I20251025 01:55:19.264559 10598 raft_consensus.cc:2468] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:19.264861 10053 catalog_manager.cc:5649] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131). New cstate: current_term: 1 leader_uuid: "90109d2fdb464942b20891b5856f8ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: HEALTHY } } }
I20251025 01:55:19.265465 10717 leader_election.cc:290] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 election: Requested vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:19.265683 10717 raft_consensus.cc:2804] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:19.265810 10717 raft_consensus.cc:697] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 LEADER]: Becoming Leader. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Running, Role: LEADER
I20251025 01:55:19.266188 10717 consensus_queue.cc:237] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.267624 10714 tablet_bootstrap.cc:492] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:19.267839 10714 ts_tablet_manager.cc:1403] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.005s
I20251025 01:55:19.268750 10714 raft_consensus.cc:359] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:19.269078 10714 raft_consensus.cc:385] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.269191 10714 raft_consensus.cc:740] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.269495 10714 consensus_queue.cc:260] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:19.270080 10714 ts_tablet_manager.cc:1434] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 01:55:19.274183 10712 tablet_bootstrap.cc:492] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:19.274348 10712 ts_tablet_manager.cc:1403] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.005s
I20251025 01:55:19.274900 10053 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_20c488ac6bf24db48ea2ab061d253e7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:19.275067 10712 raft_consensus.cc:359] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:19.275230 10712 raft_consensus.cc:385] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.275327 10712 raft_consensus.cc:740] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.275569 10712 consensus_queue.cc:260] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:19.276688 10764 raft_consensus.cc:493] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:19.276865 10764 raft_consensus.cc:515] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.277622 10764 leader_election.cc:290] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:19.278249 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8bb9f9752b641499955e678b2ccc943" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:19.278458 10598 raft_consensus.cc:2468] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:19.278919 10255 leader_election.cc:304] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.279304 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8bb9f9752b641499955e678b2ccc943" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:19.279489 10461 raft_consensus.cc:2468] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:19.279724 10713 tablet_bootstrap.cc:492] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:19.279908 10713 ts_tablet_manager.cc:1403] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.000s
I20251025 01:55:19.280604 10713 raft_consensus.cc:359] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:19.280882 10713 raft_consensus.cc:385] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.281006 10713 raft_consensus.cc:740] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.281319 10764 raft_consensus.cc:2804] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:19.281256 10713 consensus_queue.cc:260] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:19.281405 10764 raft_consensus.cc:493] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:19.281816 10764 raft_consensus.cc:3060] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.281843 10713 ts_tablet_manager.cc:1434] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 01:55:19.286198 10764 raft_consensus.cc:515] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.287068 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8bb9f9752b641499955e678b2ccc943" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:19.287297 10598 raft_consensus.cc:3060] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.290827 10598 raft_consensus.cc:2468] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:19.291460 10255 leader_election.cc:304] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.292654 10577 tablet_service.cc:1505] Processing CreateTablet for tablet db0c31aa1beb45b0936620975739ed18 (DEFAULT_TABLE table=default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.293102 10577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db0c31aa1beb45b0936620975739ed18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:19.302892 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8bb9f9752b641499955e678b2ccc943" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:19.303172 10461 raft_consensus.cc:3060] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.303661 10766 raft_consensus.cc:2804] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:19.303817 10766 raft_consensus.cc:697] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:19.304172 10766 consensus_queue.cc:237] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.307058 10461 raft_consensus.cc:2468] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:19.308480 10298 tablet_service.cc:1505] Processing CreateTablet for tablet db0c31aa1beb45b0936620975739ed18 (DEFAULT_TABLE table=default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.309958 10298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db0c31aa1beb45b0936620975739ed18. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 01:55:19.316932 10396 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.125.129:44581, user_credentials={real_user=slave}} blocked reactor thread for 60668.8us
I20251025 01:55:19.317302 10713 tablet_bootstrap.cc:492] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:19.317799 10440 tablet_service.cc:1505] Processing CreateTablet for tablet db0c31aa1beb45b0936620975739ed18 (DEFAULT_TABLE table=default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.318161 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db0c31aa1beb45b0936620975739ed18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:19.319048 10713 tablet_bootstrap.cc:654] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.319025 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f9bd9812cc74f49b4028e89d2b5f0d7" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:19.319272 10598 raft_consensus.cc:3060] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.322623 10598 raft_consensus.cc:2468] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 1.
I20251025 01:55:19.325673 10714 tablet_bootstrap.cc:492] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:19.327430 10714 tablet_bootstrap.cc:654] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.328111 10764 leader_election.cc:290] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:19.330331 10712 ts_tablet_manager.cc:1434] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.056s user 0.000s sys 0.007s
I20251025 01:55:19.330843 10712 tablet_bootstrap.cc:492] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:19.332777 10712 tablet_bootstrap.cc:654] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
W20251025 01:55:19.337595 10255 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.125.131:41021, user_credentials={real_user=slave}} blocked reactor thread for 46166.5us
I20251025 01:55:19.338802 10713 tablet_bootstrap.cc:492] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:19.339392 10713 ts_tablet_manager.cc:1403] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.009s
I20251025 01:55:19.339924 10712 tablet_bootstrap.cc:492] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:19.340054 10712 ts_tablet_manager.cc:1403] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20251025 01:55:19.340318 10714 tablet_bootstrap.cc:492] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:19.340483 10714 ts_tablet_manager.cc:1403] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.006s
I20251025 01:55:19.340785 10712 raft_consensus.cc:359] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.340953 10712 raft_consensus.cc:385] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.341039 10712 raft_consensus.cc:740] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.341259 10712 consensus_queue.cc:260] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.341364 10714 raft_consensus.cc:359] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.341631 10714 raft_consensus.cc:385] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.341764 10714 raft_consensus.cc:740] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.342059 10714 consensus_queue.cc:260] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.344343 10712 ts_tablet_manager.cc:1434] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 01:55:19.346216 10714 ts_tablet_manager.cc:1434] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251025 01:55:19.347074 10713 raft_consensus.cc:359] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.347277 10713 raft_consensus.cc:385] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.347369 10713 raft_consensus.cc:740] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.347589 10713 consensus_queue.cc:260] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.349623 10713 ts_tablet_manager.cc:1434] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251025 01:55:19.456177 10060 catalog_manager.cc:3485] renamed table default.table_ebad90bab99347a39c329ef128d4612e [id=69e063c1518a4234a260583bd0ba09c5] in HMS: new name default.table_401d389205a14ac191081b5d0df151e6
W20251025 01:55:19.466351 10237 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 consensus state has no leader
I20251025 01:55:19.466591 10769 raft_consensus.cc:493] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:19.466773 10769 raft_consensus.cc:515] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.467407 10769 leader_election.cc:290] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:19.468015 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0c31aa1beb45b0936620975739ed18" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:19.468127 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0c31aa1beb45b0936620975739ed18" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:19.468253 10321 raft_consensus.cc:2468] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 0.
I20251025 01:55:19.468318 10598 raft_consensus.cc:2468] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 0.
I20251025 01:55:19.468632 10396 leader_election.cc:304] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.469117 10769 raft_consensus.cc:2804] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:19.469224 10769 raft_consensus.cc:493] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:19.469300 10769 raft_consensus.cc:3060] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
W20251025 01:55:19.470042 10057 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cf46809300e645fca4b99a2da934b149 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:19.471496 10059 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_986ff2b14daa4beebda43706d47dc98b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 01:55:19.471967 10769 raft_consensus.cc:515] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
W20251025 01:55:19.472572 10055 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ac99b534589f45e6871ddfa3a1331b16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 01:55:19.472841 10769 leader_election.cc:290] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 election: Requested vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:19.472924 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0c31aa1beb45b0936620975739ed18" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
I20251025 01:55:19.473076 10321 raft_consensus.cc:3060] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.473572 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0c31aa1beb45b0936620975739ed18" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:19.473737 10598 raft_consensus.cc:3060] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
W20251025 01:55:19.473893 10053 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_20c488ac6bf24db48ea2ab061d253e7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 01:55:19.475638 10321 raft_consensus.cc:2468] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 1.
I20251025 01:55:19.476056 10396 leader_election.cc:304] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.476286 10598 raft_consensus.cc:2468] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 1.
I20251025 01:55:19.476362 10769 raft_consensus.cc:2804] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:19.476577 10769 raft_consensus.cc:697] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 LEADER]: Becoming Leader. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Running, Role: LEADER
I20251025 01:55:19.477037 10769 consensus_queue.cc:237] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.487450 10772 raft_consensus.cc:493] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:19.487619 10772 raft_consensus.cc:515] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.488343 10772 leader_election.cc:290] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:19.488688 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f13a1ad12e2f4f2c95103cf8d76d6da6" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:19.488871 10461 raft_consensus.cc:2468] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:19.488955 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f13a1ad12e2f4f2c95103cf8d76d6da6" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:19.489189 10598 raft_consensus.cc:2468] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:19.489320 10254 leader_election.cc:304] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.489638 10772 raft_consensus.cc:2804] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:19.489755 10772 raft_consensus.cc:493] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:19.489853 10772 raft_consensus.cc:3060] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.492676 10772 raft_consensus.cc:515] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.493597 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f13a1ad12e2f4f2c95103cf8d76d6da6" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:19.493743 10461 raft_consensus.cc:3060] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.496369 10461 raft_consensus.cc:2468] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:19.496734 10254 leader_election.cc:304] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.497043 10766 raft_consensus.cc:2804] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:19.497184 10766 raft_consensus.cc:697] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:19.497426 10766 consensus_queue.cc:237] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.500391 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f13a1ad12e2f4f2c95103cf8d76d6da6" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:19.500597 10598 raft_consensus.cc:3060] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.503154 10598 raft_consensus.cc:2468] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:19.503754 10772 leader_election.cc:290] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:19.508107 10772 raft_consensus.cc:493] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:19.508433 10772 raft_consensus.cc:515] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.509289 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7772240769e549ff8be322ce7ebc7ce4" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:19.509510 10461 raft_consensus.cc:2468] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:19.509914 10254 leader_election.cc:304] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.510213 10764 raft_consensus.cc:2804] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:19.510316 10764 raft_consensus.cc:493] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:19.510424 10764 raft_consensus.cc:3060] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.513038 10764 raft_consensus.cc:515] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
W20251025 01:55:19.513986 10046 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 consensus state has no leader
I20251025 01:55:19.514488 10764 leader_election.cc:290] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:19.515103 10462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7772240769e549ff8be322ce7ebc7ce4" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:19.515302 10462 raft_consensus.cc:3060] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.518354 10462 raft_consensus.cc:2468] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:19.519088 10254 leader_election.cc:304] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.519605 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7772240769e549ff8be322ce7ebc7ce4" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:19.519793 10598 raft_consensus.cc:3060] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.522035 10598 raft_consensus.cc:2468] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:19.522289 10764 raft_consensus.cc:2804] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:19.522660 10764 raft_consensus.cc:697] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:19.523234 10764 consensus_queue.cc:237] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.523993 10772 leader_election.cc:290] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:19.524345 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7772240769e549ff8be322ce7ebc7ce4" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:19.524736 10598 raft_consensus.cc:2376] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1. Re-sending same reply.
W20251025 01:55:19.548698 10729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:19.549090 10729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:19.549198 10729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251025 01:55:19.557206 10051 catalog_manager.cc:3485] renamed table default.table_5a6f9a84749b447180f8e258046f31ca [id=b59ac96bfe3147de99b6697bcaaed652] in HMS: new name default.table_c18eb2691b2a4c5daf9746ee47a5e451
W20251025 01:55:19.560597 10729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 01:55:19.560708 10729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:19.560817 10729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 01:55:19.560891 10729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 01:55:19.574864 10729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.188:36641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.125.188
--webserver_port=33371
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:19.577114 10729 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:19.579579 10729 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:19.597859 10786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:19.612705 10787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:19.623168 10056 catalog_manager.cc:3485] renamed table default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c] in HMS: new name default.table_328c63794848430fb0f74a446d210d94
W20251025 01:55:19.628876 10790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:19.629211 10729 server_base.cc:1047] running on GCE node
I20251025 01:55:19.630328 10729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:19.636950 10729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:19.638191 10729 hybrid_clock.cc:648] HybridClock initialized: now 1761357319638130 us; error 52 us; skew 500 ppm
I20251025 01:55:19.638597 10729 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:19.642127 10729 webserver.cc:492] Webserver started at http://127.9.125.188:33371/ using document root <none> and password file <none>
I20251025 01:55:19.642865 10729 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:19.642995 10729 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:19.651610 10769 raft_consensus.cc:493] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:19.651847 10769 raft_consensus.cc:515] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:19.652690 10769 leader_election.cc:290] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:19.653755 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcf81923a06d4750bbec550cff9bd91d" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:19.654067 10598 raft_consensus.cc:2468] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 0.
I20251025 01:55:19.654752 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcf81923a06d4750bbec550cff9bd91d" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:19.655056 10321 raft_consensus.cc:2468] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 0.
I20251025 01:55:19.655928 10396 leader_election.cc:304] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2; no voters:
I20251025 01:55:19.656608 10769 raft_consensus.cc:2804] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:19.656713 10769 raft_consensus.cc:493] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:19.656819 10769 raft_consensus.cc:3060] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.660804 10769 raft_consensus.cc:515] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:19.662055 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcf81923a06d4750bbec550cff9bd91d" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:19.662276 10598 raft_consensus.cc:3060] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.665874 10598 raft_consensus.cc:2468] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 1.
I20251025 01:55:19.666440 10396 leader_election.cc:304] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2; no voters:
I20251025 01:55:19.666785 10765 raft_consensus.cc:2804] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:19.666929 10765 raft_consensus.cc:697] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 LEADER]: Becoming Leader. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Running, Role: LEADER
I20251025 01:55:19.667857 10765 consensus_queue.cc:237] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:19.668283 10769 leader_election.cc:290] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 election: Requested vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:19.669839 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcf81923a06d4750bbec550cff9bd91d" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
I20251025 01:55:19.670081 10321 raft_consensus.cc:3060] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.671787 10729 fs_manager.cc:714] Time spent opening directory manager: real 0.027s user 0.012s sys 0.012s
I20251025 01:55:19.673385 10321 raft_consensus.cc:2468] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 1.
I20251025 01:55:19.675917 10796 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:19.678575 10729 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20251025 01:55:19.678735 10729 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
uuid: "286b4c3025e24a7c8107763fa40d3ab1"
format_stamp: "Formatted at 2025-10-25 01:55:15 on dist-test-slave-mkd1"
I20251025 01:55:19.679667 10729 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:19.683282 10054 catalog_manager.cc:3485] renamed table default.table_9e3c9d4bfaa242ce9ed8d33fc144f8be [id=91fea98dbb4548a2babcdf286454d811] in HMS: new name default.table_8553cdea51804bfa861b6c10537b99c3
I20251025 01:55:19.691696 10719 raft_consensus.cc:493] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:19.691883 10719 raft_consensus.cc:515] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.692451 10719 leader_election.cc:290] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:19.693360 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bab272a511e14adab00dd094ff1ee9e5" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:19.693379 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bab272a511e14adab00dd094ff1ee9e5" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:19.693572 10461 raft_consensus.cc:2468] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:19.693579 10321 raft_consensus.cc:2468] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:19.694008 10532 leader_election.cc:304] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.694301 10719 raft_consensus.cc:2804] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:19.694375 10719 raft_consensus.cc:493] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:19.694453 10719 raft_consensus.cc:3060] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.696935 10719 raft_consensus.cc:515] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.698137 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bab272a511e14adab00dd094ff1ee9e5" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
I20251025 01:55:19.698299 10321 raft_consensus.cc:3060] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.699990 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bab272a511e14adab00dd094ff1ee9e5" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:19.700205 10461 raft_consensus.cc:3060] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.701061 10321 raft_consensus.cc:2468] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:19.701472 10532 leader_election.cc:304] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.703052 10461 raft_consensus.cc:2468] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:19.703673 10719 leader_election.cc:290] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Requested vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:19.703907 10719 raft_consensus.cc:2804] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:19.704071 10719 raft_consensus.cc:697] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Running, Role: LEADER
I20251025 01:55:19.704387 10719 consensus_queue.cc:237] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.709873 10729 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:19.711133 10729 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:19.717304 10729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:19.720890 10057 catalog_manager.cc:2174] created HMS catalog entry for table [id=036e602729df45b8982a2ef049ec4e3a]
I20251025 01:55:19.726984 10057 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58658 (request call id 48) took 1068 ms. Trace:
I20251025 01:55:19.727123 10057 rpcz_store.cc:276] 1025 01:55:18.658832 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:18.842189 (+183357us) service_pool.cc:225] Handling call
1025 01:55:19.470154 (+627965us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 01:55:19.470693 (+ 539us) catalog_manager.cc:2147] Created new table and tablet info
1025 01:55:19.720834 (+250141us) catalog_manager.cc:2173] Created new table in HMS catalog
1025 01:55:19.726750 (+ 5916us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 01:55:19.726845 (+ 95us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 01:55:19.726945 (+ 100us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 01:55:19.721638 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 01:55:19.721768 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
1025 01:55:19.721775 (+ 7us) write_op.cc:435] Acquired schema lock
1025 01:55:19.721777 (+ 2us) tablet.cc:660] Decoding operations
1025 01:55:19.721863 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
1025 01:55:19.721877 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 01:55:19.721881 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 01:55:19.721959 (+ 78us) tablet.cc:699] Row locks acquired
1025 01:55:19.721962 (+ 3us) write_op.cc:260] PREPARE: finished
1025 01:55:19.722001 (+ 39us) write_op.cc:270] Start()
1025 01:55:19.722035 (+ 34us) write_op.cc:276] Timestamp: P: 1761357319721931 usec, L: 0
1025 01:55:19.722039 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 01:55:19.722218 (+ 179us) log.cc:844] Serialized 432 byte log entry
1025 01:55:19.725779 (+ 3561us) op_driver.cc:464] REPLICATION: finished
1025 01:55:19.725904 (+ 125us) write_op.cc:301] APPLY: starting
1025 01:55:19.725919 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 01:55:19.725967 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1025 01:55:19.725991 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 01:55:19.726115 (+ 124us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 01:55:19.726150 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 01:55:19.726157 (+ 7us) write_op.cc:312] APPLY: finished
1025 01:55:19.726229 (+ 72us) log.cc:844] Serialized 28 byte log entry
1025 01:55:19.726299 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
1025 01:55:19.726318 (+ 19us) write_op.cc:454] Released schema lock
1025 01:55:19.726324 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 01:55:19.726388 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1025 01:55:19.726391 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":212392,"Hive Metastore.run_cpu_time_us":291,"Hive Metastore.run_wall_time_us":37394,"spinlock_wait_cycles":168192,"child_traces":[["op",{"apply.queue_time_us":78,"apply.run_cpu_time_us":477,"apply.run_wall_time_us":476,"num_ops":2,"prepare.queue_time_us":78,"prepare.run_cpu_time_us":726,"prepare.run_wall_time_us":726,"raft.queue_time_us":911,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":248,"replication_time_us":3711}]]}
I20251025 01:55:19.744004 10298 tablet_service.cc:1505] Processing CreateTablet for tablet ce356b2a74154595bafdc91c27767b09 (DEFAULT_TABLE table=default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.744454 10298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce356b2a74154595bafdc91c27767b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:19.743161 10440 tablet_service.cc:1505] Processing CreateTablet for tablet ce356b2a74154595bafdc91c27767b09 (DEFAULT_TABLE table=default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.747825 10577 tablet_service.cc:1505] Processing CreateTablet for tablet ce356b2a74154595bafdc91c27767b09 (DEFAULT_TABLE table=default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.748068 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce356b2a74154595bafdc91c27767b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:19.748234 10577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce356b2a74154595bafdc91c27767b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:19.753542 10714 tablet_bootstrap.cc:492] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:19.754751 10057 catalog_manager.cc:5649] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130). New cstate: current_term: 1 leader_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20251025 01:55:19.755121 10714 tablet_bootstrap.cc:654] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.758147 10712 tablet_bootstrap.cc:492] T ce356b2a74154595bafdc91c27767b09 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:19.759694 10712 tablet_bootstrap.cc:654] T ce356b2a74154595bafdc91c27767b09 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.759984 10057 catalog_manager.cc:5741] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549) does not have the latest schema for tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]). Expected version 1 got 0
I20251025 01:55:19.772047 10057 catalog_manager.cc:5649] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b reported cstate change: term changed from 0 to 1, leader changed from <none> to c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129). New cstate: current_term: 1 leader_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20251025 01:55:19.772696 10713 tablet_bootstrap.cc:492] T ce356b2a74154595bafdc91c27767b09 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:19.774416 10713 tablet_bootstrap.cc:654] T ce356b2a74154595bafdc91c27767b09 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.786013 10058 catalog_manager.cc:3485] renamed table default.table_39f4a74b6880456585fc71ca55bff11b [id=57909f703ac64c74b92f656b0b67781f] in HMS: new name default.table_827ca3c8c36543b29dac05ab520c552a
I20251025 01:55:19.786818 10714 tablet_bootstrap.cc:492] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:19.786995 10714 ts_tablet_manager.cc:1403] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.014s
I20251025 01:55:19.787776 10714 raft_consensus.cc:359] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.787971 10714 raft_consensus.cc:385] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 01:55:19.787992 10046 catalog_manager.cc:5101] could not convert tablet's 9f9bd9812cc74f49b4028e89d2b5f0d7 schema from PB: OK
I20251025 01:55:19.788041 10714 raft_consensus.cc:740] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.791738 10714 consensus_queue.cc:260] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.796947 10714 ts_tablet_manager.cc:1434] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20251025 01:55:19.794998 10057 catalog_manager.cc:5649] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131). New cstate: current_term: 1 leader_uuid: "90109d2fdb464942b20891b5856f8ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: HEALTHY } } }
I20251025 01:55:19.812680 10598 raft_consensus.cc:1275] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Refusing update from remote peer 1f6e00cb735e4cb2bd1221cdcbc3c62c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 01:55:19.814095 10321 raft_consensus.cc:1275] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Refusing update from remote peer 1f6e00cb735e4cb2bd1221cdcbc3c62c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 01:55:19.815220 10769 consensus_queue.cc:1048] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:19.815878 10769 consensus_queue.cc:1048] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:19.817093 10712 tablet_bootstrap.cc:492] T ce356b2a74154595bafdc91c27767b09 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:19.817269 10712 ts_tablet_manager.cc:1403] T ce356b2a74154595bafdc91c27767b09 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.015s
I20251025 01:55:19.817066 10057 catalog_manager.cc:5649] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130). New cstate: current_term: 1 leader_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: HEALTHY } } }
I20251025 01:55:19.817857 10057 catalog_manager.cc:5649] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130). New cstate: current_term: 1 leader_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20251025 01:55:19.818039 10712 raft_consensus.cc:359] T ce356b2a74154595bafdc91c27767b09 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.818228 10712 raft_consensus.cc:385] T ce356b2a74154595bafdc91c27767b09 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.818302 10712 raft_consensus.cc:740] T ce356b2a74154595bafdc91c27767b09 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.818547 10712 consensus_queue.cc:260] T ce356b2a74154595bafdc91c27767b09 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.822764 10713 tablet_bootstrap.cc:492] T ce356b2a74154595bafdc91c27767b09 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:19.823644 10713 ts_tablet_manager.cc:1403] T ce356b2a74154595bafdc91c27767b09 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.007s
I20251025 01:55:19.824873 10713 raft_consensus.cc:359] T ce356b2a74154595bafdc91c27767b09 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.825102 10713 raft_consensus.cc:385] T ce356b2a74154595bafdc91c27767b09 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.825196 10713 raft_consensus.cc:740] T ce356b2a74154595bafdc91c27767b09 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.825454 10713 consensus_queue.cc:260] T ce356b2a74154595bafdc91c27767b09 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.825855 10712 ts_tablet_manager.cc:1434] T ce356b2a74154595bafdc91c27767b09 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251025 01:55:19.829064 10713 ts_tablet_manager.cc:1434] T ce356b2a74154595bafdc91c27767b09 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 01:55:19.830020 10057 catalog_manager.cc:5649] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b reported cstate change: term changed from 0 to 1, leader changed from <none> to c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129). New cstate: current_term: 1 leader_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20251025 01:55:19.830612 10057 catalog_manager.cc:5649] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b reported cstate change: term changed from 0 to 1, leader changed from <none> to c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129). New cstate: current_term: 1 leader_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20251025 01:55:19.831020 10059 catalog_manager.cc:2174] created HMS catalog entry for table [id=10c6d32941e545699b3285ec0b50f78b]
I20251025 01:55:19.831081 10057 catalog_manager.cc:5649] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b reported cstate change: term changed from 0 to 1, leader changed from <none> to c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129). New cstate: current_term: 1 leader_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20251025 01:55:19.839067 10059 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58658 (request call id 50) took 1151 ms. Trace:
I20251025 01:55:19.839195 10059 rpcz_store.cc:276] 1025 01:55:18.687283 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:18.853580 (+166297us) service_pool.cc:225] Handling call
1025 01:55:19.471597 (+618017us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 01:55:19.472022 (+ 425us) catalog_manager.cc:2147] Created new table and tablet info
1025 01:55:19.830988 (+358966us) catalog_manager.cc:2173] Created new table in HMS catalog
1025 01:55:19.838710 (+ 7722us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 01:55:19.838844 (+ 134us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 01:55:19.838992 (+ 148us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 01:55:19.832245 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 01:55:19.832400 (+ 155us) write_op.cc:432] Acquiring schema lock in shared mode
1025 01:55:19.832408 (+ 8us) write_op.cc:435] Acquired schema lock
1025 01:55:19.832413 (+ 5us) tablet.cc:660] Decoding operations
1025 01:55:19.832526 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
1025 01:55:19.832542 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1025 01:55:19.832550 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
1025 01:55:19.832652 (+ 102us) tablet.cc:699] Row locks acquired
1025 01:55:19.832657 (+ 5us) write_op.cc:260] PREPARE: finished
1025 01:55:19.832704 (+ 47us) write_op.cc:270] Start()
1025 01:55:19.832740 (+ 36us) write_op.cc:276] Timestamp: P: 1761357319832611 usec, L: 0
1025 01:55:19.832746 (+ 6us) op_driver.cc:348] REPLICATION: starting
1025 01:55:19.832945 (+ 199us) log.cc:844] Serialized 432 byte log entry
1025 01:55:19.837894 (+ 4949us) op_driver.cc:464] REPLICATION: finished
1025 01:55:19.838133 (+ 239us) write_op.cc:301] APPLY: starting
1025 01:55:19.838147 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 01:55:19.838185 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
1025 01:55:19.838206 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 01:55:19.838423 (+ 217us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 01:55:19.838457 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 01:55:19.838464 (+ 7us) write_op.cc:312] APPLY: finished
1025 01:55:19.838534 (+ 70us) log.cc:844] Serialized 28 byte log entry
1025 01:55:19.838592 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1025 01:55:19.838606 (+ 14us) write_op.cc:454] Released schema lock
1025 01:55:19.838612 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 01:55:19.838701 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
1025 01:55:19.838708 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":313791,"Hive Metastore.run_cpu_time_us":264,"Hive Metastore.run_wall_time_us":44965,"child_traces":[["op",{"apply.queue_time_us":186,"apply.run_cpu_time_us":556,"apply.run_wall_time_us":556,"num_ops":2,"prepare.queue_time_us":375,"prepare.run_cpu_time_us":826,"prepare.run_wall_time_us":826,"raft.queue_time_us":1944,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":343,"replication_time_us":5121,"spinlock_wait_cycles":59776}]]}
I20251025 01:55:19.853225 10810 mvcc.cc:204] Tried to move back new op lower bound from 7214519581942484992 to 7214519579718447104. Current Snapshot: MvccSnapshot[applied={T|T < 7214519581942484992}]
I20251025 01:55:19.863150 10819 tablet.cc:1722] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 01:55:19.870848 10809 mvcc.cc:204] Tried to move back new op lower bound from 7214519581942484992 to 7214519579718447104. Current Snapshot: MvccSnapshot[applied={T|T < 7214519581942484992}]
I20251025 01:55:19.871575 10818 tablet.cc:1722] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 01:55:19.883668 10823 tablet.cc:1722] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 01:55:19.891242 10059 catalog_manager.cc:5741] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581) does not have the latest schema for tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]). Expected version 1 got 0
I20251025 01:55:19.891683 10577 tablet_service.cc:1505] Processing CreateTablet for tablet 95491aa8141440fc999a3c742a24d503 (DEFAULT_TABLE table=default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.892155 10577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95491aa8141440fc999a3c742a24d503. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:19.892411 10059 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_083845c9d31747a68c31d8a1804a12aa" } modify_external_catalogs: true
I20251025 01:55:19.892611 10059 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_083845c9d31747a68c31d8a1804a12aa" } modify_external_catalogs: true
I20251025 01:55:19.893380 10057 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_a981c91dba574e1c8ccce55a41c4bd88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:19.897800 10440 tablet_service.cc:1505] Processing CreateTablet for tablet 95491aa8141440fc999a3c742a24d503 (DEFAULT_TABLE table=default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.898186 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95491aa8141440fc999a3c742a24d503. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:19.902724 10298 tablet_service.cc:1505] Processing CreateTablet for tablet 95491aa8141440fc999a3c742a24d503 (DEFAULT_TABLE table=default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.903113 10298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95491aa8141440fc999a3c742a24d503. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:19.904186 10712 tablet_bootstrap.cc:492] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:19.904651 10055 catalog_manager.cc:2174] created HMS catalog entry for table [id=df0ae3390867456e81e7f7856272b2a2]
I20251025 01:55:19.905257 10714 tablet_bootstrap.cc:492] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:19.905781 10712 tablet_bootstrap.cc:654] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.906843 10714 tablet_bootstrap.cc:654] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.910275 10055 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58658 (request call id 51) took 1162 ms. Trace:
I20251025 01:55:19.910399 10055 rpcz_store.cc:276] 1025 01:55:18.748240 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:19.149343 (+401103us) service_pool.cc:225] Handling call
1025 01:55:19.472667 (+323324us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 01:55:19.473062 (+ 395us) catalog_manager.cc:2147] Created new table and tablet info
1025 01:55:19.904620 (+431558us) catalog_manager.cc:2173] Created new table in HMS catalog
1025 01:55:19.909924 (+ 5304us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 01:55:19.910081 (+ 157us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 01:55:19.910206 (+ 125us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 01:55:19.905294 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 01:55:19.905400 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
1025 01:55:19.905405 (+ 5us) write_op.cc:435] Acquired schema lock
1025 01:55:19.905407 (+ 2us) tablet.cc:660] Decoding operations
1025 01:55:19.905489 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
1025 01:55:19.905503 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 01:55:19.905507 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 01:55:19.905578 (+ 71us) tablet.cc:699] Row locks acquired
1025 01:55:19.905581 (+ 3us) write_op.cc:260] PREPARE: finished
1025 01:55:19.905614 (+ 33us) write_op.cc:270] Start()
1025 01:55:19.905645 (+ 31us) write_op.cc:276] Timestamp: P: 1761357319905509 usec, L: 0
1025 01:55:19.905648 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 01:55:19.905818 (+ 170us) log.cc:844] Serialized 432 byte log entry
1025 01:55:19.908793 (+ 2975us) op_driver.cc:464] REPLICATION: finished
1025 01:55:19.909278 (+ 485us) write_op.cc:301] APPLY: starting
1025 01:55:19.909296 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 01:55:19.909358 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
1025 01:55:19.909388 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 01:55:19.909548 (+ 160us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 01:55:19.909621 (+ 73us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 01:55:19.909632 (+ 11us) write_op.cc:312] APPLY: finished
1025 01:55:19.909718 (+ 86us) log.cc:844] Serialized 28 byte log entry
1025 01:55:19.909786 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1025 01:55:19.909810 (+ 24us) write_op.cc:454] Released schema lock
1025 01:55:19.909819 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1025 01:55:19.909907 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
1025 01:55:19.909914 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":357788,"Hive Metastore.run_cpu_time_us":265,"Hive Metastore.run_wall_time_us":73622,"child_traces":[["op",{"apply.queue_time_us":422,"apply.run_cpu_time_us":619,"apply.run_wall_time_us":620,"num_ops":2,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":637,"prepare.run_wall_time_us":637,"raft.queue_time_us":158,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":297,"replication_time_us":3111}]]}
I20251025 01:55:19.914863 10714 tablet_bootstrap.cc:492] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:19.914994 10714 ts_tablet_manager.cc:1403] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20251025 01:55:19.915736 10714 raft_consensus.cc:359] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:19.915906 10714 raft_consensus.cc:385] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.915985 10714 raft_consensus.cc:740] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.916200 10714 consensus_queue.cc:260] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:19.918391 10714 ts_tablet_manager.cc:1434] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 01:55:19.923058 10712 tablet_bootstrap.cc:492] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:19.923199 10712 ts_tablet_manager.cc:1403] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.009s
I20251025 01:55:19.923920 10712 raft_consensus.cc:359] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:19.924067 10712 raft_consensus.cc:385] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.924137 10712 raft_consensus.cc:740] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.924371 10712 consensus_queue.cc:260] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:19.924660 10055 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5] alter complete (version 1)
I20251025 01:55:19.925714 10712 ts_tablet_manager.cc:1434] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 01:55:19.927479 10440 tablet_service.cc:1505] Processing CreateTablet for tablet b3578b8a1d8f412d9a8b2b968bb9c583 (DEFAULT_TABLE table=default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.927881 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3578b8a1d8f412d9a8b2b968bb9c583. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:19.928634 10577 tablet_service.cc:1505] Processing CreateTablet for tablet b3578b8a1d8f412d9a8b2b968bb9c583 (DEFAULT_TABLE table=default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.928682 10055 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "default.Table_ac8050cb49d54bc2a444d81cf8492c75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:19.928962 10577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3578b8a1d8f412d9a8b2b968bb9c583. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:19.934509 10714 tablet_bootstrap.cc:492] T b3578b8a1d8f412d9a8b2b968bb9c583 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:19.936053 10714 tablet_bootstrap.cc:654] T b3578b8a1d8f412d9a8b2b968bb9c583 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.942510 10719 raft_consensus.cc:493] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:19.942644 10719 raft_consensus.cc:515] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.943058 10301 tablet_service.cc:1505] Processing CreateTablet for tablet b3578b8a1d8f412d9a8b2b968bb9c583 (DEFAULT_TABLE table=default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:19.943287 10719 leader_election.cc:290] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:19.943578 10301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3578b8a1d8f412d9a8b2b968bb9c583. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:19.944046 10712 tablet_bootstrap.cc:492] T b3578b8a1d8f412d9a8b2b968bb9c583 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:19.943980 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce356b2a74154595bafdc91c27767b09" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:19.944072 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce356b2a74154595bafdc91c27767b09" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:19.944181 10321 raft_consensus.cc:2468] T ce356b2a74154595bafdc91c27767b09 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:19.944257 10461 raft_consensus.cc:2468] T ce356b2a74154595bafdc91c27767b09 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:19.944638 10532 leader_election.cc:304] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.945047 10719 raft_consensus.cc:2804] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:19.945159 10719 raft_consensus.cc:493] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:19.945240 10719 raft_consensus.cc:3060] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.945680 10712 tablet_bootstrap.cc:654] T b3578b8a1d8f412d9a8b2b968bb9c583 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.948331 10719 raft_consensus.cc:515] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.949209 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce356b2a74154595bafdc91c27767b09" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
I20251025 01:55:19.949399 10321 raft_consensus.cc:3060] T ce356b2a74154595bafdc91c27767b09 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.950544 10713 tablet_bootstrap.cc:492] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:19.952220 10713 tablet_bootstrap.cc:654] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.952481 10321 raft_consensus.cc:2468] T ce356b2a74154595bafdc91c27767b09 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:19.952979 10532 leader_election.cc:304] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:19.953284 10801 raft_consensus.cc:2804] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:19.954070 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce356b2a74154595bafdc91c27767b09" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:19.954301 10461 raft_consensus.cc:3060] T ce356b2a74154595bafdc91c27767b09 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:19.957434 10461 raft_consensus.cc:2468] T ce356b2a74154595bafdc91c27767b09 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:19.957939 10719 leader_election.cc:290] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Requested vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:19.958037 10712 tablet_bootstrap.cc:492] T b3578b8a1d8f412d9a8b2b968bb9c583 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:19.958189 10712 ts_tablet_manager.cc:1403] T b3578b8a1d8f412d9a8b2b968bb9c583 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.001s
I20251025 01:55:19.958971 10712 raft_consensus.cc:359] T b3578b8a1d8f412d9a8b2b968bb9c583 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.959488 10712 raft_consensus.cc:385] T b3578b8a1d8f412d9a8b2b968bb9c583 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.959606 10712 raft_consensus.cc:740] T b3578b8a1d8f412d9a8b2b968bb9c583 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.959923 10712 consensus_queue.cc:260] T b3578b8a1d8f412d9a8b2b968bb9c583 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.963101 10712 ts_tablet_manager.cc:1434] T b3578b8a1d8f412d9a8b2b968bb9c583 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 01:55:19.966897 10714 tablet_bootstrap.cc:492] T b3578b8a1d8f412d9a8b2b968bb9c583 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:19.967083 10714 ts_tablet_manager.cc:1403] T b3578b8a1d8f412d9a8b2b968bb9c583 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.000s
I20251025 01:55:19.968029 10714 raft_consensus.cc:359] T b3578b8a1d8f412d9a8b2b968bb9c583 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.968250 10714 raft_consensus.cc:385] T b3578b8a1d8f412d9a8b2b968bb9c583 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.968343 10714 raft_consensus.cc:740] T b3578b8a1d8f412d9a8b2b968bb9c583 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.968600 10714 consensus_queue.cc:260] T b3578b8a1d8f412d9a8b2b968bb9c583 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.969064 10714 ts_tablet_manager.cc:1434] T b3578b8a1d8f412d9a8b2b968bb9c583 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 01:55:19.974915 10801 raft_consensus.cc:697] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Running, Role: LEADER
I20251025 01:55:19.975383 10801 consensus_queue.cc:237] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.981084 10713 tablet_bootstrap.cc:492] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:19.981241 10713 ts_tablet_manager.cc:1403] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.007s
I20251025 01:55:19.981877 10713 raft_consensus.cc:359] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.982004 10713 raft_consensus.cc:385] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.982082 10713 raft_consensus.cc:740] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.982311 10713 consensus_queue.cc:260] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:19.986534 10713 ts_tablet_manager.cc:1434] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 01:55:19.989290 10713 tablet_bootstrap.cc:492] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:19.990885 10713 tablet_bootstrap.cc:654] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:19.996249 10713 tablet_bootstrap.cc:492] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:19.996413 10713 ts_tablet_manager.cc:1403] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20251025 01:55:19.997053 10713 raft_consensus.cc:359] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:19.997233 10713 raft_consensus.cc:385] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:19.997304 10713 raft_consensus.cc:740] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:19.997550 10713 consensus_queue.cc:260] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:20.000936 10713 ts_tablet_manager.cc:1434] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 01:55:20.042665 10766 consensus_queue.cc:1048] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.067552 10854 raft_consensus.cc:493] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:20.067737 10854 raft_consensus.cc:515] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:20.068718 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3578b8a1d8f412d9a8b2b968bb9c583" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:20.069057 10461 raft_consensus.cc:2468] T b3578b8a1d8f412d9a8b2b968bb9c583 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:20.069612 10254 leader_election.cc:304] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:20.076670 10850 raft_consensus.cc:2804] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:20.076781 10850 raft_consensus.cc:493] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:20.076908 10850 raft_consensus.cc:3060] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:20.079322 10766 consensus_queue.cc:1048] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.084721 10850 raft_consensus.cc:515] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:20.085754 10597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3578b8a1d8f412d9a8b2b968bb9c583" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:20.085963 10597 raft_consensus.cc:3060] T b3578b8a1d8f412d9a8b2b968bb9c583 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:20.088604 10597 raft_consensus.cc:2468] T b3578b8a1d8f412d9a8b2b968bb9c583 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:20.089141 10255 leader_election.cc:304] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:20.089464 10856 raft_consensus.cc:2804] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:20.089653 10856 raft_consensus.cc:697] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:20.090006 10856 consensus_queue.cc:237] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:20.091777 10854 leader_election.cc:290] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:20.092991 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3578b8a1d8f412d9a8b2b968bb9c583" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:20.093235 10598 raft_consensus.cc:2376] T b3578b8a1d8f412d9a8b2b968bb9c583 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1. Re-sending same reply.
I20251025 01:55:20.095916 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3578b8a1d8f412d9a8b2b968bb9c583" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:20.096287 10461 raft_consensus.cc:3060] T b3578b8a1d8f412d9a8b2b968bb9c583 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:20.102149 10850 leader_election.cc:290] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:20.103214 10461 raft_consensus.cc:2468] T b3578b8a1d8f412d9a8b2b968bb9c583 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:20.123710 10856 raft_consensus.cc:493] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:20.123883 10856 raft_consensus.cc:515] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:20.124686 10856 leader_election.cc:290] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:20.125619 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95491aa8141440fc999a3c742a24d503" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:20.125873 10461 raft_consensus.cc:2468] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:20.126699 10597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95491aa8141440fc999a3c742a24d503" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:20.126969 10597 raft_consensus.cc:2468] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:20.127720 10255 leader_election.cc:304] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:20.128909 10856 raft_consensus.cc:2804] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:20.129014 10856 raft_consensus.cc:493] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:20.129137 10856 raft_consensus.cc:3060] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:20.132851 10856 raft_consensus.cc:515] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:20.134001 10597 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95491aa8141440fc999a3c742a24d503" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:20.134245 10597 raft_consensus.cc:3060] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:20.137771 10597 raft_consensus.cc:2468] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:20.138283 10255 leader_election.cc:304] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:20.138589 10850 raft_consensus.cc:2804] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:20.138744 10850 raft_consensus.cc:697] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:20.139055 10850 consensus_queue.cc:237] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:20.143896 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95491aa8141440fc999a3c742a24d503" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:20.144146 10461 raft_consensus.cc:3060] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:20.157434 10461 raft_consensus.cc:2468] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:20.160172 10856 leader_election.cc:290] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:20.175724 10801 consensus_queue.cc:1048] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.193655 10801 consensus_queue.cc:1048] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.214061 10729 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.188:36641
I20251025 01:55:20.215976 10729 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/info.pb
I20251025 01:55:20.214085 10883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.188:36641 every 8 connection(s)
I20251025 01:55:20.225013 10884 sys_catalog.cc:263] Verifying existing consensus state
I20251025 01:55:20.226020 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 10729
I20251025 01:55:20.241870 10884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Bootstrap starting.
I20251025 01:55:20.255153 10884 log.cc:826] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:20.258514 10856 consensus_queue.cc:1048] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.266135 10884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251025 01:55:20.266826 10884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Bootstrap complete.
I20251025 01:55:20.270009 10507 heartbeater.cc:344] Connected to a master server at 127.9.125.188:36641
I20251025 01:55:20.270215 10507 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
I20251025 01:55:20.281738 10854 consensus_queue.cc:1048] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.284160 10884 raft_consensus.cc:359] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:20.286830 10884 raft_consensus.cc:740] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286b4c3025e24a7c8107763fa40d3ab1, State: Initialized, Role: FOLLOWER
I20251025 01:55:20.287734 10884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:20.289079 10507 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
I20251025 01:55:20.290062 10854 consensus_queue.cc:1048] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.298393 10890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:20.298684 10890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:20.300928 10884 sys_catalog.cc:565] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 01:55:20.307411 10507 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
I20251025 01:55:20.307940 10643 heartbeater.cc:344] Connected to a master server at 127.9.125.188:36641
I20251025 01:55:20.308161 10643 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
I20251025 01:55:20.321113 10643 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
I20251025 01:55:20.322291 10856 consensus_queue.cc:1048] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.325230 10643 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
I20251025 01:55:20.331481 10369 heartbeater.cc:344] Connected to a master server at 127.9.125.188:36641
I20251025 01:55:20.331692 10643 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
I20251025 01:55:20.335350 10507 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
I20251025 01:55:20.374294 10908 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:20.374406 10908 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: loading cluster ID for follower catalog manager: success
I20251025 01:55:20.392408 10908 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: acquiring CA information for follower catalog manager: success
I20251025 01:55:20.396529 10908 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 01:55:20.448647 10856 consensus_queue.cc:1048] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.449851 10643 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
I20251025 01:55:20.459055 10831 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" instance_seqno: 1761357317567829) as {username='slave'} at 127.9.125.131:33771; Asking this server to re-register.
I20251025 01:55:20.460103 10643 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:20.462209 10831 ts_manager.cc:194] Registered new tserver with Master: 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:20.468075 10852 consensus_queue.cc:1048] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.469460 10507 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
I20251025 01:55:20.472704 10801 consensus_queue.cc:1048] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.472808 10913 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:55:20.472977 10913 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:20.473623 10913 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:20.474210 10068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 44 } } ignore_live_leader: false dest_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" is_pre_election: true
I20251025 01:55:20.478060 10769 consensus_queue.cc:1048] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.482043 10836 consensus_queue.cc:1048] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.484887 10847 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 44 } } ignore_live_leader: false dest_uuid: "286b4c3025e24a7c8107763fa40d3ab1" is_pre_election: true
I20251025 01:55:20.485491 10847 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76559300dae44c0d8a17b583a8875b3f in term 1.
I20251025 01:55:20.486042 9974 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce
I20251025 01:55:20.488708 10831 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" instance_seqno: 1761357316719619) as {username='slave'} at 127.9.125.129:38955; Asking this server to re-register.
I20251025 01:55:20.492867 10369 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:20.493992 10913 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 01:55:20.494159 10913 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:55:20.494338 10913 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 2 FOLLOWER]: Advancing to term 3
I20251025 01:55:20.494613 10831 ts_manager.cc:194] Registered new tserver with Master: c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:20.497440 10913 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:20.498610 10068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 44 } } ignore_live_leader: false dest_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce"
I20251025 01:55:20.499720 10847 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 44 } } ignore_live_leader: false dest_uuid: "286b4c3025e24a7c8107763fa40d3ab1"
I20251025 01:55:20.499883 10847 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 1 FOLLOWER]: Advancing to term 3
I20251025 01:55:20.506733 10847 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76559300dae44c0d8a17b583a8875b3f in term 3.
I20251025 01:55:20.506904 10913 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 3 election: Requested vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:20.508724 10000 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 3 FOLLOWER]: Rejecting Update request from peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce for earlier term 2. Current term is 3. Ops: []
I20251025 01:55:20.508927 9974 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce
I20251025 01:55:20.509619 10694 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 }, Status: INVALID_TERM, Last received: 2.44, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251025 01:55:20.509766 10913 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 3 FOLLOWER]: Leader election won for term 3
I20251025 01:55:20.553788 10922 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 2 LEADER]: Stepping down as leader of term 2
I20251025 01:55:20.555178 10922 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, State: Running, Role: LEADER
I20251025 01:55:20.555841 10922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 2.44, Last appended by leader: 44, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:20.556514 10922 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 2 FOLLOWER]: Advancing to term 3
I20251025 01:55:20.561668 10913 raft_consensus.cc:697] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 3 LEADER]: Becoming Leader. State: Replica: 76559300dae44c0d8a17b583a8875b3f, State: Running, Role: LEADER
I20251025 01:55:20.562283 10913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 2.44, Last appended by leader: 44, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:20.563350 10831 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" instance_seqno: 1761357317153916) as {username='slave'} at 127.9.125.130:60595; Asking this server to re-register.
I20251025 01:55:20.564568 10507 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:20.566329 10831 ts_manager.cc:194] Registered new tserver with Master: 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:20.567031 10923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76559300dae44c0d8a17b583a8875b3f. Latest consensus state: current_term: 3 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:20.567457 10923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:20.571360 10926 consensus_queue.cc:1048] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.576861 10927 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 01:55:20.592574 10927 catalog_manager.cc:679] Loaded metadata for table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a]
I20251025 01:55:20.595840 10927 catalog_manager.cc:679] Loaded metadata for table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b]
I20251025 01:55:20.596450 10927 catalog_manager.cc:679] Loaded metadata for table default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c]
I20251025 01:55:20.596870 10927 catalog_manager.cc:679] Loaded metadata for table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]
I20251025 01:55:20.597277 10927 catalog_manager.cc:679] Loaded metadata for table default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956]
I20251025 01:55:20.597708 10927 catalog_manager.cc:679] Loaded metadata for table default.table_39f4a74b6880456585fc71ca55bff11b [id=57909f703ac64c74b92f656b0b67781f]
I20251025 01:55:20.598109 10927 catalog_manager.cc:679] Loaded metadata for table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]
I20251025 01:55:20.598564 10927 catalog_manager.cc:679] Loaded metadata for table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]
I20251025 01:55:20.598961 10927 catalog_manager.cc:679] Loaded metadata for table default.table_a2d327beb950460cbf080c4c44d89943 [id=7477ea6c70f84bc2897964f18294d49f]
I20251025 01:55:20.599411 10927 catalog_manager.cc:679] Loaded metadata for table default.table_9e3c9d4bfaa242ce9ed8d33fc144f8be [id=91fea98dbb4548a2babcdf286454d811]
I20251025 01:55:20.599828 10927 catalog_manager.cc:679] Loaded metadata for table default.table_5a6f9a84749b447180f8e258046f31ca [id=b59ac96bfe3147de99b6697bcaaed652]
I20251025 01:55:20.600246 10927 catalog_manager.cc:679] Loaded metadata for table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2]
I20251025 01:55:20.612540 10927 tablet_loader.cc:96] loaded metadata for tablet 47b9e247969945439741dea4298b99ab (table default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956])
I20251025 01:55:20.613060 10927 tablet_loader.cc:96] loaded metadata for tablet 7772240769e549ff8be322ce7ebc7ce4 (table default.table_39f4a74b6880456585fc71ca55bff11b [id=57909f703ac64c74b92f656b0b67781f])
I20251025 01:55:20.613924 10927 tablet_loader.cc:96] loaded metadata for tablet 95491aa8141440fc999a3c742a24d503 (table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b])
I20251025 01:55:20.614573 10927 tablet_loader.cc:96] loaded metadata for tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5])
I20251025 01:55:20.614919 10927 tablet_loader.cc:96] loaded metadata for tablet a8bb9f9752b641499955e678b2ccc943 (table default.table_9e3c9d4bfaa242ce9ed8d33fc144f8be [id=91fea98dbb4548a2babcdf286454d811])
I20251025 01:55:20.615330 10927 tablet_loader.cc:96] loaded metadata for tablet b3578b8a1d8f412d9a8b2b968bb9c583 (table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2])
I20251025 01:55:20.615710 10927 tablet_loader.cc:96] loaded metadata for tablet bab272a511e14adab00dd094ff1ee9e5 (table default.table_a2d327beb950460cbf080c4c44d89943 [id=7477ea6c70f84bc2897964f18294d49f])
I20251025 01:55:20.616050 10927 tablet_loader.cc:96] loaded metadata for tablet ce356b2a74154595bafdc91c27767b09 (table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a])
I20251025 01:55:20.616420 10927 tablet_loader.cc:96] loaded metadata for tablet db0c31aa1beb45b0936620975739ed18 (table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b])
I20251025 01:55:20.616782 10927 tablet_loader.cc:96] loaded metadata for tablet dcf81923a06d4750bbec550cff9bd91d (table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812])
I20251025 01:55:20.617156 10927 tablet_loader.cc:96] loaded metadata for tablet df08f7a1b9ca4b8e8328d9295bf52576 (table default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c])
I20251025 01:55:20.617520 10927 tablet_loader.cc:96] loaded metadata for tablet f13a1ad12e2f4f2c95103cf8d76d6da6 (table default.table_5a6f9a84749b447180f8e258046f31ca [id=b59ac96bfe3147de99b6697bcaaed652])
I20251025 01:55:20.617810 10927 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 01:55:20.619299 10927 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:20.619383 10927 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 01:55:20.621138 10927 catalog_manager.cc:1514] Loading token signing keys...
I20251025 01:55:20.622426 10927 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Loaded TSK: 0
I20251025 01:55:20.623533 10927 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 01:55:20.624694 10927 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251025 01:55:20.625945 10927 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: 11
I20251025 01:55:20.667977 10052 catalog_manager.cc:2731] deleted HMS catalog entry for table default.table_a2d327beb950460cbf080c4c44d89943 [id=7477ea6c70f84bc2897964f18294d49f]
I20251025 01:55:20.694301 10053 catalog_manager.cc:2174] created HMS catalog entry for table [id=3ffd4312df4f4b6f8ce3b4ecc4da08c1]
W20251025 01:55:20.695415 10053 catalog_manager.cc:2202] Illegal state: an error occurred while writing to the sys-catalog: replica 0ceaf0e1ce0f4c23b39a5bf4736de5ce is not leader of this config: current role FOLLOWER
W20251025 01:55:20.701759 10237 catalog_manager.cc:3922] Illegal state: an error occurred while updating the sys-catalog: replica 0ceaf0e1ce0f4c23b39a5bf4736de5ce is not leader of this config: current role FOLLOWER
W20251025 01:55:20.702029 10237 hms_notification_log_listener.cc:330] Failed to handle Hive Metastore notification: 12 ALTER_TABLE default.table_c18eb2691b2a4c5daf9746ee47a5e451: Illegal state: an error occurred while updating the sys-catalog: replica 0ceaf0e1ce0f4c23b39a5bf4736de5ce is not leader of this config: current role FOLLOWER
W20251025 01:55:20.702163 10237 hms_notification_log_listener.cc:143] Hive Metastore notification log listener poll failed: Service unavailable: lost leadership while handling Hive Metastore notification log events: an error occurred while updating the sys-catalog: replica 0ceaf0e1ce0f4c23b39a5bf4736de5ce is not leader of this config: current role FOLLOWER
I20251025 01:55:20.702656 10060 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:58658 (request call id 39) took 2245 ms. Trace:
I20251025 01:55:20.702749 10060 rpcz_store.cc:276] 1025 01:55:18.457070 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:18.457318 (+ 248us) service_pool.cc:225] Handling call
1025 01:55:19.283906 (+826588us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 01:55:20.702599 (+1418693us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":49,"Hive Metastore.run_cpu_time_us":1027,"Hive Metastore.run_wall_time_us":171794}
W20251025 01:55:20.703998 10060 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.9.125.130:49409 (request call id 12) took 774 ms (client timeout 1000 ms). Trace:
W20251025 01:55:20.704085 10060 rpcz_store.cc:269] 1025 01:55:19.929568 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:20.703115 (+773547us) service_pool.cc:225] Handling call
1025 01:55:20.703946 (+ 831us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 01:55:20.704231 10651 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.125.189:45553)
W20251025 01:55:20.705035 10060 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.9.125.131:48027 (request call id 12) took 772 ms (client timeout 1000 ms). Trace:
W20251025 01:55:20.705117 10060 rpcz_store.cc:269] 1025 01:55:19.932499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:20.704306 (+771807us) service_pool.cc:225] Handling call
1025 01:55:20.704986 (+ 680us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 01:55:20.705925 10060 master_service.cc:954] ReplaceTablet: received request to replace tablet 7772240769e549ff8be322ce7ebc7ce4 from {username='slave'} at 127.0.0.1:58658
I20251025 01:55:20.706207 10060 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:58658 (request call id 57) took 1692 ms. Trace:
I20251025 01:55:20.706301 10060 rpcz_store.cc:276] 1025 01:55:19.013295 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:20.705859 (+1692564us) service_pool.cc:225] Handling call
1025 01:55:20.706177 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 01:55:20.706523 10060 master_service.cc:954] ReplaceTablet: received request to replace tablet 47b9e247969945439741dea4298b99ab from {username='slave'} at 127.0.0.1:58658
I20251025 01:55:20.706785 10060 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:58658 (request call id 58) took 1692 ms. Trace:
I20251025 01:55:20.706849 10060 rpcz_store.cc:276] 1025 01:55:19.014177 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:20.706483 (+1692306us) service_pool.cc:225] Handling call
1025 01:55:20.706744 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 01:55:20.707036 10060 master_service.cc:954] ReplaceTablet: received request to replace tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 from {username='slave'} at 127.0.0.1:58658
I20251025 01:55:20.709189 10051 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:58658 (request call id 40) took 2221 ms. Trace:
I20251025 01:55:20.709311 10051 rpcz_store.cc:276] 1025 01:55:18.487295 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:18.487356 (+ 61us) service_pool.cc:225] Handling call
1025 01:55:19.468736 (+981380us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 01:55:20.708403 (+1239667us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 01:55:20.708859 (+ 456us) catalog_manager.cc:3708] Apply alter schema
1025 01:55:20.709131 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":89,"Hive Metastore.run_cpu_time_us":680,"Hive Metastore.run_wall_time_us":87986,"spinlock_wait_cycles":2304}
I20251025 01:55:20.709198 10054 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:58658 (request call id 42) took 2127 ms. Trace:
I20251025 01:55:20.710059 10051 master_service.cc:954] ReplaceTablet: received request to replace tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 from {username='slave'} at 127.0.0.1:58658
I20251025 01:55:20.710397 10054 rpcz_store.cc:276] 1025 01:55:18.581932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:18.582060 (+ 128us) service_pool.cc:225] Handling call
1025 01:55:19.469508 (+887448us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 01:55:20.708679 (+1239171us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 01:55:20.708906 (+ 227us) catalog_manager.cc:3708] Apply alter schema
1025 01:55:20.709143 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":153338,"Hive Metastore.run_cpu_time_us":554,"Hive Metastore.run_wall_time_us":60175}
I20251025 01:55:20.710933 10056 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:58658 (request call id 41) took 2162 ms. Trace:
I20251025 01:55:20.711035 10056 rpcz_store.cc:276] 1025 01:55:18.548241 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:18.548288 (+ 47us) service_pool.cc:225] Handling call
1025 01:55:19.469195 (+920907us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 01:55:20.708388 (+1239193us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 01:55:20.710693 (+ 2305us) catalog_manager.cc:3708] Apply alter schema
1025 01:55:20.710892 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":87684,"Hive Metastore.run_cpu_time_us":662,"Hive Metastore.run_wall_time_us":65866}
W20251025 01:55:20.711966 10651 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.125.189:45553)
I20251025 01:55:20.712217 10056 master_service.cc:954] ReplaceTablet: received request to replace tablet df08f7a1b9ca4b8e8328d9295bf52576 from {username='slave'} at 127.0.0.1:58658
I20251025 01:55:20.712531 10056 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:58658 (request call id 61) took 1664 ms. Trace:
I20251025 01:55:20.712653 10054 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58658 (request call id 62) took 1655 ms. Trace:
I20251025 01:55:20.712785 10054 rpcz_store.cc:276] 1025 01:55:19.057413 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:20.712353 (+1654940us) service_pool.cc:225] Handling call
1025 01:55:20.712617 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 01:55:20.713177 10051 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:58658 (request call id 60) took 1664 ms. Trace:
I20251025 01:55:20.713297 10051 rpcz_store.cc:276] 1025 01:55:19.045858 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:20.709588 (+1663730us) service_pool.cc:225] Handling call
1025 01:55:20.710322 (+ 734us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 01:55:20.713776 10051 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:58658 (request call id 64) took 1494 ms. Trace:
I20251025 01:55:20.713856 10051 rpcz_store.cc:276] 1025 01:55:19.219122 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:20.713468 (+1494346us) service_pool.cc:225] Handling call
1025 01:55:20.713726 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 01:55:20.714211 9990 master_service.cc:954] ReplaceTablet: received request to replace tablet 47b9e247969945439741dea4298b99ab from {username='slave'} at 127.0.0.1:40888
I20251025 01:55:20.714293 10051 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:58658 (request call id 65) took 1446 ms. Trace:
I20251025 01:55:20.714370 10051 rpcz_store.cc:276] 1025 01:55:19.267632 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:20.714054 (+1446422us) service_pool.cc:225] Handling call
1025 01:55:20.714251 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 01:55:20.714229 9988 master_service.cc:954] ReplaceTablet: received request to replace tablet 7772240769e549ff8be322ce7ebc7ce4 from {username='slave'} at 127.0.0.1:40888
I20251025 01:55:20.714761 10051 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58658 (request call id 66) took 1295 ms. Trace:
I20251025 01:55:20.714833 10051 rpcz_store.cc:276] 1025 01:55:19.419116 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:20.714502 (+1295386us) service_pool.cc:225] Handling call
1025 01:55:20.714718 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 01:55:20.712625 10056 rpcz_store.cc:276] 1025 01:55:19.048107 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:20.712163 (+1664056us) service_pool.cc:225] Handling call
1025 01:55:20.712477 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 01:55:20.715884 10921 consensus_queue.cc:1048] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.717784 10058 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:58658 (request call id 49) took 2046 ms. Trace:
I20251025 01:55:20.717902 10058 rpcz_store.cc:276] 1025 01:55:18.670858 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:18.843969 (+173111us) service_pool.cc:225] Handling call
1025 01:55:19.470997 (+627028us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 01:55:20.717319 (+1246322us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 01:55:20.717523 (+ 204us) catalog_manager.cc:3708] Apply alter schema
1025 01:55:20.717743 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":249536,"Hive Metastore.run_cpu_time_us":637,"Hive Metastore.run_wall_time_us":65216}
W20251025 01:55:20.718762 10057 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a981c91dba574e1c8ccce55a41c4bd88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:20.719868 10055 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ac8050cb49d54bc2a444d81cf8492c75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 01:55:20.720697 10052 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:58658 (request call id 52) took 1938 ms. Trace:
I20251025 01:55:20.720777 10052 rpcz_store.cc:276] 1025 01:55:18.782596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:19.242867 (+460271us) service_pool.cc:225] Handling call
1025 01:55:19.473468 (+230601us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 01:55:20.720647 (+1247179us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":430961,"Hive Metastore.run_cpu_time_us":302,"Hive Metastore.run_wall_time_us":763276}
I20251025 01:55:20.713171 10054 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58658 (request call id 63) took 1557 ms. Trace:
I20251025 01:55:20.721305 10054 rpcz_store.cc:276] 1025 01:55:19.155291 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:20.712911 (+1557620us) service_pool.cc:225] Handling call
1025 01:55:20.713121 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 01:55:20.721529 10051 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58658 (request call id 67) took 1252 ms. Trace:
I20251025 01:55:20.721623 10729 master_runner.cc:232] Existing masters have differing terms: 2 vs 3
I20251025 01:55:20.721740 10051 rpcz_store.cc:276] 1025 01:55:19.468734 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:20.714958 (+1246224us) service_pool.cc:225] Handling call
1025 01:55:20.721483 (+ 6525us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 01:55:20.721853 10729 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:20.722007 10060 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:58658 (request call id 59) took 1677 ms. Trace:
I20251025 01:55:20.722142 10060 rpcz_store.cc:276] 1025 01:55:19.044381 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:20.706970 (+1662589us) service_pool.cc:225] Handling call
1025 01:55:20.721976 (+ 15006us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 01:55:20.723395 9989 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_ebad90bab99347a39c329ef128d4612e" } new_table_name: "default.Table_401d389205a14ac191081b5d0df151e6" modify_external_catalogs: true
W20251025 01:55:20.731106 10651 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.125.190:35925)
W20251025 01:55:20.734550 10651 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.125.190:35925)
I20251025 01:55:20.736153 10847 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 3 FOLLOWER]: Refusing update from remote peer 76559300dae44c0d8a17b583a8875b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 46. (index mismatch)
I20251025 01:55:20.736852 10923 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 4, Time since last communication: 0.000s
I20251025 01:55:20.743933 10068 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 3 FOLLOWER]: Refusing update from remote peer 76559300dae44c0d8a17b583a8875b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 44. Preceding OpId from leader: term: 3 index: 46. (index mismatch)
I20251025 01:55:20.744503 10923 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251025 01:55:20.747874 10370 heartbeater.cc:499] Master 127.9.125.190:35925 was elected leader, sending a full tablet report...
I20251025 01:55:20.765631 10924 consensus_queue.cc:1048] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.767068 10922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76559300dae44c0d8a17b583a8875b3f. Latest consensus state: current_term: 3 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:20.767294 10922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:20.805051 9981 master_service.cc:954] ReplaceTablet: received request to replace tablet df08f7a1b9ca4b8e8328d9295bf52576 from {username='slave'} at 127.0.0.1:40888
I20251025 01:55:20.804776 9983 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_44e4ccc4f2de47198a0599e2577108cf" } modify_external_catalogs: true
I20251025 01:55:20.805655 9983 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_44e4ccc4f2de47198a0599e2577108cf" } modify_external_catalogs: true
I20251025 01:55:20.805972 10644 heartbeater.cc:499] Master 127.9.125.190:35925 was elected leader, sending a full tablet report...
I20251025 01:55:20.806161 9982 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40888:
name: "default.Table_9187de27a70f48e08cc7284131fe543d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:20.807543 9985 master_service.cc:954] ReplaceTablet: received request to replace tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 from {username='slave'} at 127.0.0.1:40888
I20251025 01:55:20.808688 9984 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_19b5f67b2cfa4e62bc0d061b1c2c8ecf" } modify_external_catalogs: true
I20251025 01:55:20.808908 9984 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_19b5f67b2cfa4e62bc0d061b1c2c8ecf" } modify_external_catalogs: true
I20251025 01:55:20.808895 9986 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40888:
name: "default.Table_547c7e192439428aa78c62d503e17536"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:20.809535 10913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:20.809737 10913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:20.811095 10053 catalog_manager.cc:2184] deleted orphaned HMS catalog entry for table [id=3ffd4312df4f4b6f8ce3b4ecc4da08c1]
I20251025 01:55:20.811422 10053 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58658 (request call id 53) took 2028 ms. Trace:
I20251025 01:55:20.811555 10053 rpcz_store.cc:276] 1025 01:55:18.782711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:19.274705 (+491994us) service_pool.cc:225] Handling call
1025 01:55:19.473973 (+199268us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 01:55:19.474352 (+ 379us) catalog_manager.cc:2147] Created new table and tablet info
1025 01:55:20.694238 (+1219886us) catalog_manager.cc:2173] Created new table in HMS catalog
1025 01:55:20.695160 (+ 922us) write_op.cc:489] Releasing partition, row and schema locks
1025 01:55:20.695164 (+ 4us) write_op.cc:454] Released schema lock
1025 01:55:20.695522 (+ 358us) catalog_manager.cc:2181] Rolling back HMS table creation
1025 01:55:20.811363 (+115841us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 01:55:20.695003 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0ceaf0e1ce0f4c23b39a5bf4736de5ce is not leader of this config: current role FOLLOWER)
1025 01:55:20.695018 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
1025 01:55:20.695026 (+ 8us) write_op.cc:454] Released schema lock
1025 01:55:20.695031 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1197011,"Hive Metastore.run_cpu_time_us":481,"Hive Metastore.run_wall_time_us":137955,"child_traces":[["op",{}]]}
I20251025 01:55:20.812453 10922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:20.812633 10922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:20.824991 10934 mvcc.cc:204] Tried to move back new op lower bound from 7214519585726832640 to 7214519585027862528. Current Snapshot: MvccSnapshot[applied={T|T < 7214519585726832640 or (T in {7214519585726832640})}]
I20251025 01:55:20.828943 9988 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Sending DeleteTablet for 3 replicas of tablet 7772240769e549ff8be322ce7ebc7ce4
I20251025 01:55:20.832446 9988 catalog_manager.cc:6719] ReplaceTablet: tablet 7772240769e549ff8be322ce7ebc7ce4 deleted and replaced by tablet cceaea14d4054de09cdaacb93f9b5582
I20251025 01:55:20.854254 9990 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Sending DeleteTablet for 3 replicas of tablet 47b9e247969945439741dea4298b99ab
I20251025 01:55:20.854770 9990 catalog_manager.cc:6719] ReplaceTablet: tablet 47b9e247969945439741dea4298b99ab deleted and replaced by tablet 8aa21b52709149139ddb4f99d9ba228c
I20251025 01:55:20.859927 10301 tablet_service.cc:1552] Processing DeleteTablet for tablet 7772240769e549ff8be322ce7ebc7ce4 with delete_type TABLET_DATA_DELETED (replaced by tablet cceaea14d4054de09cdaacb93f9b5582) from {username='slave'} at 127.0.0.1:49356
I20251025 01:55:20.860100 10298 tablet_service.cc:1552] Processing DeleteTablet for tablet 47b9e247969945439741dea4298b99ab with delete_type TABLET_DATA_DELETED (replaced by tablet 8aa21b52709149139ddb4f99d9ba228c) from {username='slave'} at 127.0.0.1:49356
W20251025 01:55:20.861990 9976 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.125.189:45553, user_credentials={real_user=slave}} blocked reactor thread for 43736.4us
I20251025 01:55:20.863533 10577 tablet_service.cc:1552] Processing DeleteTablet for tablet 7772240769e549ff8be322ce7ebc7ce4 with delete_type TABLET_DATA_DELETED (replaced by tablet cceaea14d4054de09cdaacb93f9b5582) from {username='slave'} at 127.0.0.1:43660
I20251025 01:55:20.863767 10578 tablet_service.cc:1552] Processing DeleteTablet for tablet 47b9e247969945439741dea4298b99ab with delete_type TABLET_DATA_DELETED (replaced by tablet 8aa21b52709149139ddb4f99d9ba228c) from {username='slave'} at 127.0.0.1:43660
I20251025 01:55:20.865589 9990 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_cf46809300e645fca4b99a2da934b149" } modify_external_catalogs: true
I20251025 01:55:20.865831 9990 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_cf46809300e645fca4b99a2da934b149" } modify_external_catalogs: true
I20251025 01:55:20.868758 10946 tablet_replica.cc:333] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:20.869032 10946 raft_consensus.cc:2243] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:20.869737 10946 raft_consensus.cc:2272] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:20.871295 9981 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Sending DeleteTablet for 3 replicas of tablet df08f7a1b9ca4b8e8328d9295bf52576
I20251025 01:55:20.871793 9981 catalog_manager.cc:6719] ReplaceTablet: tablet df08f7a1b9ca4b8e8328d9295bf52576 deleted and replaced by tablet 0541393def0b4dcc9cf86038e01d1c21
I20251025 01:55:20.872227 9985 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Sending DeleteTablet for 3 replicas of tablet 9f9bd9812cc74f49b4028e89d2b5f0d7
I20251025 01:55:20.872709 9985 catalog_manager.cc:6719] ReplaceTablet: tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 deleted and replaced by tablet b8f17943e2d0462da38dcdfb4b1cc025
I20251025 01:55:20.872828 10298 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 with delete_type TABLET_DATA_DELETED (replaced by tablet b8f17943e2d0462da38dcdfb4b1cc025) from {username='slave'} at 127.0.0.1:49356
I20251025 01:55:20.873001 10301 tablet_service.cc:1552] Processing DeleteTablet for tablet df08f7a1b9ca4b8e8328d9295bf52576 with delete_type TABLET_DATA_DELETED (replaced by tablet 0541393def0b4dcc9cf86038e01d1c21) from {username='slave'} at 127.0.0.1:49356
I20251025 01:55:20.873656 10578 tablet_service.cc:1552] Processing DeleteTablet for tablet df08f7a1b9ca4b8e8328d9295bf52576 with delete_type TABLET_DATA_DELETED (replaced by tablet 0541393def0b4dcc9cf86038e01d1c21) from {username='slave'} at 127.0.0.1:43660
I20251025 01:55:20.874019 10578 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 with delete_type TABLET_DATA_DELETED (replaced by tablet b8f17943e2d0462da38dcdfb4b1cc025) from {username='slave'} at 127.0.0.1:43660
I20251025 01:55:20.874607 9987 catalog_manager.cc:5649] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b reported cstate change: term changed from 0 to 1, leader changed from <none> to c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129). New cstate: current_term: 1 leader_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20251025 01:55:20.876390 9987 catalog_manager.cc:5649] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b reported cstate change: term changed from 0 to 1, leader changed from <none> to c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129). New cstate: current_term: 1 leader_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: UNKNOWN } } }
I20251025 01:55:20.888628 10947 tablet_replica.cc:333] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:20.889317 10946 ts_tablet_manager.cc:1916] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:20.889979 10440 tablet_service.cc:1552] Processing DeleteTablet for tablet 7772240769e549ff8be322ce7ebc7ce4 with delete_type TABLET_DATA_DELETED (replaced by tablet cceaea14d4054de09cdaacb93f9b5582) from {username='slave'} at 127.0.0.1:58374
I20251025 01:55:20.894510 10441 tablet_service.cc:1552] Processing DeleteTablet for tablet 47b9e247969945439741dea4298b99ab with delete_type TABLET_DATA_DELETED (replaced by tablet 8aa21b52709149139ddb4f99d9ba228c) from {username='slave'} at 127.0.0.1:58374
I20251025 01:55:20.894765 10442 tablet_service.cc:1552] Processing DeleteTablet for tablet df08f7a1b9ca4b8e8328d9295bf52576 with delete_type TABLET_DATA_DELETED (replaced by tablet 0541393def0b4dcc9cf86038e01d1c21) from {username='slave'} at 127.0.0.1:58374
I20251025 01:55:20.895231 10442 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 with delete_type TABLET_DATA_DELETED (replaced by tablet b8f17943e2d0462da38dcdfb4b1cc025) from {username='slave'} at 127.0.0.1:58374
I20251025 01:55:20.897158 10947 raft_consensus.cc:2243] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:20.902890 10947 raft_consensus.cc:2272] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:20.898627 10852 consensus_queue.cc:1048] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.922658 10947 ts_tablet_manager.cc:1916] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:20.924299 9981 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40888:
name: "default.Table_d0ff04fccfe8480d9d96d87f94272591"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:20.924973 9985 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40888:
name: "default.Table_6b948e527acb484d8c82d7ea153e9304"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:20.926020 10949 tablet_replica.cc:333] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:20.926232 10949 raft_consensus.cc:2243] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:20.926465 10949 raft_consensus.cc:2272] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:20.930533 10949 ts_tablet_manager.cc:1916] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:20.932513 10059 catalog_manager.cc:2731] deleted HMS catalog entry for table default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956]
I20251025 01:55:20.932951 10059 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:58658 (request call id 54) took 2072 ms. Trace:
I20251025 01:55:20.933044 10059 rpcz_store.cc:276] 1025 01:55:18.860185 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:19.892269 (+1032084us) service_pool.cc:225] Handling call
1025 01:55:20.718237 (+825968us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 01:55:20.932892 (+214655us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":92550,"Hive Metastore.run_cpu_time_us":174,"Hive Metastore.run_wall_time_us":121508}
I20251025 01:55:20.936203 10947 ts_tablet_manager.cc:1929] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:20.936332 10947 log.cc:1199] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/7772240769e549ff8be322ce7ebc7ce4
I20251025 01:55:20.936772 10947 ts_tablet_manager.cc:1950] T 7772240769e549ff8be322ce7ebc7ce4 P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:20.939594 10946 ts_tablet_manager.cc:1929] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:20.939715 10946 log.cc:1199] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/7772240769e549ff8be322ce7ebc7ce4
I20251025 01:55:20.940138 10946 ts_tablet_manager.cc:1950] T 7772240769e549ff8be322ce7ebc7ce4 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:20.940766 10850 consensus_queue.cc:1048] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:20.940996 10946 tablet_replica.cc:333] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:20.941182 10946 raft_consensus.cc:2243] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:20.941341 10946 raft_consensus.cc:2272] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:20.942198 10508 heartbeater.cc:499] Master 127.9.125.190:35925 was elected leader, sending a full tablet report...
I20251025 01:55:20.942430 10946 ts_tablet_manager.cc:1916] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:20.948314 10949 ts_tablet_manager.cc:1929] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:20.948426 10949 log.cc:1199] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/7772240769e549ff8be322ce7ebc7ce4
I20251025 01:55:20.948808 10949 ts_tablet_manager.cc:1950] T 7772240769e549ff8be322ce7ebc7ce4 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:20.949239 10949 tablet_replica.cc:333] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:20.949415 10949 raft_consensus.cc:2243] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:20.949576 10949 raft_consensus.cc:2272] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:20.955749 10949 ts_tablet_manager.cc:1916] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:20.956279 10946 ts_tablet_manager.cc:1929] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:20.956403 10946 log.cc:1199] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/47b9e247969945439741dea4298b99ab
I20251025 01:55:20.956775 10946 ts_tablet_manager.cc:1950] T 47b9e247969945439741dea4298b99ab P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:20.957640 10946 tablet_replica.cc:333] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:20.957824 10946 raft_consensus.cc:2243] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:20.958073 10946 raft_consensus.cc:2272] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:20.958660 10946 ts_tablet_manager.cc:1916] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:20.961953 10949 ts_tablet_manager.cc:1929] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:20.962077 10949 log.cc:1199] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/47b9e247969945439741dea4298b99ab
I20251025 01:55:20.962502 10949 ts_tablet_manager.cc:1950] T 47b9e247969945439741dea4298b99ab P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:20.963469 9975 catalog_manager.cc:4985] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): tablet 7772240769e549ff8be322ce7ebc7ce4 (table default.table_39f4a74b6880456585fc71ca55bff11b [id=57909f703ac64c74b92f656b0b67781f]) successfully deleted
I20251025 01:55:20.964383 10947 tablet_replica.cc:333] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:20.964608 10947 raft_consensus.cc:2243] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:20.964993 9975 catalog_manager.cc:4985] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): tablet 47b9e247969945439741dea4298b99ab (table default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956]) successfully deleted
I20251025 01:55:20.965088 10947 raft_consensus.cc:2272] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:20.965245 9974 catalog_manager.cc:4985] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): tablet 7772240769e549ff8be322ce7ebc7ce4 (table default.table_39f4a74b6880456585fc71ca55bff11b [id=57909f703ac64c74b92f656b0b67781f]) successfully deleted
I20251025 01:55:20.965906 9975 catalog_manager.cc:4985] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): tablet 7772240769e549ff8be322ce7ebc7ce4 (table default.table_39f4a74b6880456585fc71ca55bff11b [id=57909f703ac64c74b92f656b0b67781f]) successfully deleted
I20251025 01:55:20.967583 10947 ts_tablet_manager.cc:1916] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:20.972716 9974 catalog_manager.cc:4985] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): tablet 47b9e247969945439741dea4298b99ab (table default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956]) successfully deleted
I20251025 01:55:20.973812 9988 catalog_manager.cc:5649] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131). New cstate: current_term: 1 leader_uuid: "90109d2fdb464942b20891b5856f8ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: HEALTHY } } }
I20251025 01:55:20.974831 10577 tablet_service.cc:1505] Processing CreateTablet for tablet 8aa21b52709149139ddb4f99d9ba228c (DEFAULT_TABLE table=default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:20.975078 10301 tablet_service.cc:1505] Processing CreateTablet for tablet 8aa21b52709149139ddb4f99d9ba228c (DEFAULT_TABLE table=default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 01:55:20.975149 10204 catalog_manager.cc:5101] could not convert tablet's f13a1ad12e2f4f2c95103cf8d76d6da6 schema from PB: OK
I20251025 01:55:20.975203 10577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa21b52709149139ddb4f99d9ba228c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:20.976202 10298 tablet_service.cc:1505] Processing CreateTablet for tablet cceaea14d4054de09cdaacb93f9b5582 (DEFAULT_TABLE table=default.table_39f4a74b6880456585fc71ca55bff11b [id=57909f703ac64c74b92f656b0b67781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:20.976881 10949 tablet_replica.cc:333] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:20.977063 10949 raft_consensus.cc:2243] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:20.977241 10949 raft_consensus.cc:2272] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:20.977995 10949 ts_tablet_manager.cc:1916] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:20.980556 10297 tablet_service.cc:1552] Processing DeleteTablet for tablet 47b9e247969945439741dea4298b99ab with delete_type TABLET_DATA_DELETED (replaced by tablet 8aa21b52709149139ddb4f99d9ba228c) from {username='slave'} at 127.0.0.1:49356
I20251025 01:55:20.981503 10300 tablet_service.cc:1552] Processing DeleteTablet for tablet 7772240769e549ff8be322ce7ebc7ce4 with delete_type TABLET_DATA_DELETED (replaced by tablet cceaea14d4054de09cdaacb93f9b5582) from {username='slave'} at 127.0.0.1:49356
I20251025 01:55:20.982474 10578 tablet_service.cc:1505] Processing CreateTablet for tablet cceaea14d4054de09cdaacb93f9b5582 (DEFAULT_TABLE table=default.table_39f4a74b6880456585fc71ca55bff11b [id=57909f703ac64c74b92f656b0b67781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:20.982805 10578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cceaea14d4054de09cdaacb93f9b5582. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:20.989763 10957 tablet_bootstrap.cc:492] T cceaea14d4054de09cdaacb93f9b5582 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:20.993326 10057 catalog_manager.cc:2174] created HMS catalog entry for table [id=269cfd64fbd94f0486843525917d9bb4]
W20251025 01:55:20.993997 10057 catalog_manager.cc:2202] Illegal state: an error occurred while writing to the sys-catalog: replica 0ceaf0e1ce0f4c23b39a5bf4736de5ce is not leader of this config: current role FOLLOWER
I20251025 01:55:20.997822 10957 tablet_bootstrap.cc:654] T cceaea14d4054de09cdaacb93f9b5582 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:20.998433 10301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa21b52709149139ddb4f99d9ba228c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:20.998488 10946 ts_tablet_manager.cc:1929] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 01:55:20.998637 10946 log.cc:1199] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/9f9bd9812cc74f49b4028e89d2b5f0d7
I20251025 01:55:20.998968 10298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cceaea14d4054de09cdaacb93f9b5582. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:20.999118 10946 ts_tablet_manager.cc:1950] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:21.000185 9975 catalog_manager.cc:4985] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]) successfully deleted
I20251025 01:55:21.001188 9987 catalog_manager.cc:5741] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549) does not have the latest schema for tablet f13a1ad12e2f4f2c95103cf8d76d6da6 (table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652]). Expected version 1 got 0
I20251025 01:55:21.003126 10576 tablet_service.cc:1505] Processing CreateTablet for tablet b8f17943e2d0462da38dcdfb4b1cc025 (DEFAULT_TABLE table=default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.003527 10576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f17943e2d0462da38dcdfb4b1cc025. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.008339 10300 tablet_service.cc:1505] Processing CreateTablet for tablet 0541393def0b4dcc9cf86038e01d1c21 (DEFAULT_TABLE table=default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.008450 9988 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40888:
name: "default.Table_dfca34fd2dec49089ebdfe61dcc20ba8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:21.008757 10300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0541393def0b4dcc9cf86038e01d1c21. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 01:55:21.009109 10204 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0541393def0b4dcc9cf86038e01d1c21 consensus state has no leader
I20251025 01:55:21.011541 10299 tablet_service.cc:1505] Processing CreateTablet for tablet b8f17943e2d0462da38dcdfb4b1cc025 (DEFAULT_TABLE table=default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.011934 10299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f17943e2d0462da38dcdfb4b1cc025. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.019134 10577 tablet_service.cc:1552] Processing DeleteTablet for tablet 7772240769e549ff8be322ce7ebc7ce4 with delete_type TABLET_DATA_DELETED (replaced by tablet cceaea14d4054de09cdaacb93f9b5582) from {username='slave'} at 127.0.0.1:43660
I20251025 01:55:21.028090 10442 tablet_service.cc:1505] Processing CreateTablet for tablet cceaea14d4054de09cdaacb93f9b5582 (DEFAULT_TABLE table=default.table_39f4a74b6880456585fc71ca55bff11b [id=57909f703ac64c74b92f656b0b67781f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 01:55:21.034343 9977 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 0541393def0b4dcc9cf86038e01d1c21 (table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0541393def0b4dcc9cf86038e01d1c21 consensus state has no leader
I20251025 01:55:21.035742 10442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cceaea14d4054de09cdaacb93f9b5582. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.036077 10440 tablet_service.cc:1505] Processing CreateTablet for tablet 8aa21b52709149139ddb4f99d9ba228c (DEFAULT_TABLE table=default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.035742 10441 tablet_service.cc:1505] Processing CreateTablet for tablet 0541393def0b4dcc9cf86038e01d1c21 (DEFAULT_TABLE table=default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.036566 10438 tablet_service.cc:1552] Processing DeleteTablet for tablet 7772240769e549ff8be322ce7ebc7ce4 with delete_type TABLET_DATA_DELETED (replaced by tablet cceaea14d4054de09cdaacb93f9b5582) from {username='slave'} at 127.0.0.1:58374
I20251025 01:55:21.037420 10578 tablet_service.cc:1505] Processing CreateTablet for tablet 0541393def0b4dcc9cf86038e01d1c21 (DEFAULT_TABLE table=default.table_437718d96b1e4760a51380e0f777933e [id=3df0b36649ab414c9c552abbd401d14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.037806 10578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0541393def0b4dcc9cf86038e01d1c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.038223 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa21b52709149139ddb4f99d9ba228c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.038769 10437 tablet_service.cc:1552] Processing DeleteTablet for tablet 47b9e247969945439741dea4298b99ab with delete_type TABLET_DATA_DELETED (replaced by tablet 8aa21b52709149139ddb4f99d9ba228c) from {username='slave'} at 127.0.0.1:58374
W20251025 01:55:21.039213 10204 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cceaea14d4054de09cdaacb93f9b5582 consensus state has no leader
I20251025 01:55:21.039698 10439 tablet_service.cc:1505] Processing CreateTablet for tablet b8f17943e2d0462da38dcdfb4b1cc025 (DEFAULT_TABLE table=default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.041440 10204 catalog_manager.cc:2772] Deleting table default.table_a2d327beb950460cbf080c4c44d89943 [id=7477ea6c70f84bc2897964f18294d49f] in response to Hive Metastore notification log event 19
I20251025 01:55:21.041683 10441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0541393def0b4dcc9cf86038e01d1c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.046281 10960 tablet.cc:1722] T f13a1ad12e2f4f2c95103cf8d76d6da6 P c3e4f9f2253e416e8b31ea86ead23a3b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 01:55:21.047596 9987 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40888:
name: "default.Table_31d2075af6df48048aa516b629e1caa3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:21.048789 10204 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Sending DeleteTablet for 3 replicas of tablet bab272a511e14adab00dd094ff1ee9e5
I20251025 01:55:21.050115 10300 tablet_service.cc:1552] Processing DeleteTablet for tablet bab272a511e14adab00dd094ff1ee9e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 01:55:21 UTC) from {username='slave'} at 127.0.0.1:49356
I20251025 01:55:21.050175 10959 tablet_bootstrap.cc:492] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:21.051040 10576 tablet_service.cc:1552] Processing DeleteTablet for tablet bab272a511e14adab00dd094ff1ee9e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 01:55:21 UTC) from {username='slave'} at 127.0.0.1:43660
I20251025 01:55:21.053620 10946 tablet_replica.cc:333] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:21.053892 10439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f17943e2d0462da38dcdfb4b1cc025. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.055564 10836 consensus_queue.cc:1048] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:21.063474 10946 raft_consensus.cc:2243] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:21.063783 10946 raft_consensus.cc:2272] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.064309 10946 ts_tablet_manager.cc:1916] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.066843 10955 tablet.cc:1722] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 01:55:21.067822 10963 tablet_bootstrap.cc:492] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:21.069502 10963 tablet_bootstrap.cc:654] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.071307 9989 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40888:
name: "default.Table_b90ae6ea0cd54c2cb1ad07289907c3c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:21.071415 10956 tablet.cc:1722] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 90109d2fdb464942b20891b5856f8ac2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 01:55:21.078058 10947 ts_tablet_manager.cc:1929] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:21.078190 10947 log.cc:1199] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/47b9e247969945439741dea4298b99ab
I20251025 01:55:21.078581 10947 ts_tablet_manager.cc:1950] T 47b9e247969945439741dea4298b99ab P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:21.079595 10965 tablet.cc:1722] T f13a1ad12e2f4f2c95103cf8d76d6da6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 01:55:21.080750 10966 tablet.cc:1722] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 01:55:21.081590 10850 consensus_queue.cc:1048] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:21.083299 10949 ts_tablet_manager.cc:1929] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:21.083603 10949 log.cc:1199] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/df08f7a1b9ca4b8e8328d9295bf52576
I20251025 01:55:21.083953 10959 tablet_bootstrap.cc:654] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.083487 10440 tablet_service.cc:1552] Processing DeleteTablet for tablet bab272a511e14adab00dd094ff1ee9e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 01:55:21 UTC) from {username='slave'} at 127.0.0.1:58374
I20251025 01:55:21.084290 10949 ts_tablet_manager.cc:1950] T df08f7a1b9ca4b8e8328d9295bf52576 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:21.085115 10949 tablet_replica.cc:333] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:21.085350 9974 catalog_manager.cc:4985] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): tablet df08f7a1b9ca4b8e8328d9295bf52576 (table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c]) successfully deleted
I20251025 01:55:21.085858 10949 raft_consensus.cc:2243] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:21.086428 10949 raft_consensus.cc:2272] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.087076 10949 ts_tablet_manager.cc:1916] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.093154 10850 consensus_queue.cc:1048] T b3578b8a1d8f412d9a8b2b968bb9c583 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:21.097669 10946 ts_tablet_manager.cc:1929] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:21.097816 10946 log.cc:1199] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/df08f7a1b9ca4b8e8328d9295bf52576
I20251025 01:55:21.100986 10957 tablet_bootstrap.cc:492] T cceaea14d4054de09cdaacb93f9b5582 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:21.101197 10957 ts_tablet_manager.cc:1403] T cceaea14d4054de09cdaacb93f9b5582 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.112s user 0.007s sys 0.006s
I20251025 01:55:21.102289 10957 raft_consensus.cc:359] T cceaea14d4054de09cdaacb93f9b5582 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.102680 10957 raft_consensus.cc:385] T cceaea14d4054de09cdaacb93f9b5582 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.102818 10957 raft_consensus.cc:740] T cceaea14d4054de09cdaacb93f9b5582 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.103161 10957 consensus_queue.cc:260] T cceaea14d4054de09cdaacb93f9b5582 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.104789 10946 ts_tablet_manager.cc:1950] T df08f7a1b9ca4b8e8328d9295bf52576 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:21.105204 10957 ts_tablet_manager.cc:1434] T cceaea14d4054de09cdaacb93f9b5582 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 01:55:21.105546 10949 ts_tablet_manager.cc:1929] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 01:55:21.105703 10949 log.cc:1199] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/9f9bd9812cc74f49b4028e89d2b5f0d7
I20251025 01:55:21.105834 9975 catalog_manager.cc:4985] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): tablet df08f7a1b9ca4b8e8328d9295bf52576 (table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c]) successfully deleted
I20251025 01:55:21.106068 10949 ts_tablet_manager.cc:1950] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:21.107187 9974 catalog_manager.cc:4985] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]) successfully deleted
W20251025 01:55:21.107671 9975 catalog_manager.cc:4952] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): delete failed for tablet 47b9e247969945439741dea4298b99ab because the tablet was not found. No further retry: Not found: Tablet not found: 47b9e247969945439741dea4298b99ab
W20251025 01:55:21.108625 9975 catalog_manager.cc:4952] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): delete failed for tablet 7772240769e549ff8be322ce7ebc7ce4 because the tablet was not found. No further retry: Not found: Tablet not found: 7772240769e549ff8be322ce7ebc7ce4
I20251025 01:55:21.109107 10946 tablet_replica.cc:333] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:21.109295 10946 raft_consensus.cc:2243] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:21.109521 10946 raft_consensus.cc:2272] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.109807 10957 tablet_bootstrap.cc:492] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
W20251025 01:55:21.111430 9974 catalog_manager.cc:4952] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): delete failed for tablet 7772240769e549ff8be322ce7ebc7ce4 because the tablet was not found. No further retry: Not found: Tablet not found: 7772240769e549ff8be322ce7ebc7ce4
W20251025 01:55:21.115744 9974 catalog_manager.cc:4952] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): delete failed for tablet 47b9e247969945439741dea4298b99ab because the tablet was not found. No further retry: Not found: Tablet not found: 47b9e247969945439741dea4298b99ab
I20251025 01:55:21.115872 10946 ts_tablet_manager.cc:1916] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.116127 10949 tablet_replica.cc:333] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:21.116340 10949 raft_consensus.cc:2243] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:21.116556 10949 raft_consensus.cc:2272] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.116747 10204 catalog_manager.cc:2772] Deleting table default.table_20c488ac6bf24db48ea2ab061d253e7c [id=3ffd4312df4f4b6f8ce3b4ecc4da08c1] in response to Hive Metastore notification log event 21
W20251025 01:55:21.117102 10204 hms_notification_log_listener.cc:330] Failed to handle Hive Metastore notification: 21 DROP_TABLE default.table_20c488ac6bf24db48ea2ab061d253e7c: Not found: the table does not exist: table_id: "3ffd4312df4f4b6f8ce3b4ecc4da08c1" table_name: "default.table_20c488ac6bf24db48ea2ab061d253e7c"
I20251025 01:55:21.118322 10957 tablet_bootstrap.cc:654] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.118948 10949 ts_tablet_manager.cc:1916] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.121562 10959 tablet_bootstrap.cc:492] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:21.122933 10956 tablet.cc:1722] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 01:55:21.124033 10947 tablet_replica.cc:333] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:21.124429 9975 catalog_manager.cc:4985] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): tablet 47b9e247969945439741dea4298b99ab (table default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956]) successfully deleted
I20251025 01:55:21.125030 10947 raft_consensus.cc:2243] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:21.125245 10947 raft_consensus.cc:2272] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.125725 10947 ts_tablet_manager.cc:1916] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.126312 10959 ts_tablet_manager.cc:1403] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.000s
I20251025 01:55:21.127687 10959 raft_consensus.cc:359] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.127961 10959 raft_consensus.cc:385] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.128170 10959 raft_consensus.cc:740] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.128760 10959 consensus_queue.cc:260] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.128965 10055 catalog_manager.cc:2174] created HMS catalog entry for table [id=2f021b58f5304a3f8e7d46b745a7165a]
I20251025 01:55:21.129289 10959 ts_tablet_manager.cc:1434] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251025 01:55:21.129791 10055 catalog_manager.cc:2202] Illegal state: an error occurred while writing to the sys-catalog: replica 0ceaf0e1ce0f4c23b39a5bf4736de5ce is not leader of this config: current role FOLLOWER
I20251025 01:55:21.130435 10959 tablet_bootstrap.cc:492] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:21.136667 10959 tablet_bootstrap.cc:654] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.138248 10890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76559300dae44c0d8a17b583a8875b3f. Latest consensus state: current_term: 3 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:21.138490 10890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:21.139755 10204 catalog_manager.cc:2772] Deleting table default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956] in response to Hive Metastore notification log event 22
I20251025 01:55:21.139897 10952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:21.142694 10963 tablet_bootstrap.cc:492] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:21.142841 10963 ts_tablet_manager.cc:1403] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.000s
I20251025 01:55:21.143036 10952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:21.143453 10938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:21.143766 10938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:21.146337 10963 raft_consensus.cc:359] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.146508 10963 raft_consensus.cc:385] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.146587 10963 raft_consensus.cc:740] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.151815 10204 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Sending DeleteTablet for 3 replicas of tablet 8aa21b52709149139ddb4f99d9ba228c
I20251025 01:55:21.151751 10836 consensus_queue.cc:1048] T ce356b2a74154595bafdc91c27767b09 P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:21.152971 10440 tablet_service.cc:1552] Processing DeleteTablet for tablet 8aa21b52709149139ddb4f99d9ba228c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 01:55:21 UTC) from {username='slave'} at 127.0.0.1:58374
I20251025 01:55:21.153395 10578 tablet_service.cc:1552] Processing DeleteTablet for tablet 8aa21b52709149139ddb4f99d9ba228c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 01:55:21 UTC) from {username='slave'} at 127.0.0.1:43660
I20251025 01:55:21.154340 10301 tablet_service.cc:1552] Processing DeleteTablet for tablet 8aa21b52709149139ddb4f99d9ba228c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 01:55:21 UTC) from {username='slave'} at 127.0.0.1:49356
I20251025 01:55:21.154713 10963 consensus_queue.cc:260] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.156085 10963 ts_tablet_manager.cc:1434] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251025 01:55:21.156385 10963 tablet_bootstrap.cc:492] T cceaea14d4054de09cdaacb93f9b5582 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:21.157971 10963 tablet_bootstrap.cc:654] T cceaea14d4054de09cdaacb93f9b5582 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.158044 10949 ts_tablet_manager.cc:1929] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:21.158181 10949 log.cc:1199] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/bab272a511e14adab00dd094ff1ee9e5
I20251025 01:55:21.158667 10949 ts_tablet_manager.cc:1950] T bab272a511e14adab00dd094ff1ee9e5 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:21.159350 10952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:21.159512 10952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:21.164887 10947 ts_tablet_manager.cc:1929] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:21.165020 10947 log.cc:1199] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/df08f7a1b9ca4b8e8328d9295bf52576
I20251025 01:55:21.165407 10947 ts_tablet_manager.cc:1950] T df08f7a1b9ca4b8e8328d9295bf52576 P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:21.166280 9975 catalog_manager.cc:4985] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): tablet df08f7a1b9ca4b8e8328d9295bf52576 (table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c]) successfully deleted
I20251025 01:55:21.166566 10947 tablet_replica.cc:333] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:21.166761 10947 raft_consensus.cc:2243] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:21.166994 10947 raft_consensus.cc:2272] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.167519 10947 ts_tablet_manager.cc:1916] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.168372 9974 catalog_manager.cc:4985] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): tablet bab272a511e14adab00dd094ff1ee9e5 (table default.table_a2d327beb950460cbf080c4c44d89943 [id=7477ea6c70f84bc2897964f18294d49f]) successfully deleted
I20251025 01:55:21.168937 10949 tablet_replica.cc:333] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:21.169152 10949 raft_consensus.cc:2243] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:21.169317 10949 raft_consensus.cc:2272] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.173619 10949 ts_tablet_manager.cc:1916] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.176525 10957 tablet_bootstrap.cc:492] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:21.176687 10957 ts_tablet_manager.cc:1403] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.002s
I20251025 01:55:21.177449 10957 raft_consensus.cc:359] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.177629 10957 raft_consensus.cc:385] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.177707 10957 raft_consensus.cc:740] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
W20251025 01:55:21.177814 9982 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9187de27a70f48e08cc7284131fe543d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 01:55:21.177989 10957 consensus_queue.cc:260] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.178529 10957 ts_tablet_manager.cc:1434] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251025 01:55:21.179814 9986 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_547c7e192439428aa78c62d503e17536 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:21.179917 9981 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d0ff04fccfe8480d9d96d87f94272591 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:21.180804 9985 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6b948e527acb484d8c82d7ea153e9304 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:21.181794 9988 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:21.182600 9987 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_31d2075af6df48048aa516b629e1caa3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 01:55:21.183444 9989 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 01:55:21.190819 10946 ts_tablet_manager.cc:1929] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:21.197165 10963 tablet_bootstrap.cc:492] T cceaea14d4054de09cdaacb93f9b5582 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:21.197336 10963 ts_tablet_manager.cc:1403] T cceaea14d4054de09cdaacb93f9b5582 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20251025 01:55:21.198172 10963 raft_consensus.cc:359] T cceaea14d4054de09cdaacb93f9b5582 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.198361 10963 raft_consensus.cc:385] T cceaea14d4054de09cdaacb93f9b5582 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.198439 10963 raft_consensus.cc:740] T cceaea14d4054de09cdaacb93f9b5582 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.198722 10963 consensus_queue.cc:260] T cceaea14d4054de09cdaacb93f9b5582 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.199951 10963 ts_tablet_manager.cc:1434] T cceaea14d4054de09cdaacb93f9b5582 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 01:55:21.200258 10963 tablet_bootstrap.cc:492] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:21.200770 10949 ts_tablet_manager.cc:1929] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:21.200943 10949 log.cc:1199] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/8aa21b52709149139ddb4f99d9ba228c
I20251025 01:55:21.201365 10949 ts_tablet_manager.cc:1950] T 8aa21b52709149139ddb4f99d9ba228c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:21.201908 10963 tablet_bootstrap.cc:654] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.202797 10957 tablet_bootstrap.cc:492] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:21.204496 9974 catalog_manager.cc:4985] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): tablet 8aa21b52709149139ddb4f99d9ba228c (table default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956]) successfully deleted
I20251025 01:55:21.208181 10957 tablet_bootstrap.cc:654] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
W20251025 01:55:21.210383 10531 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.125.130:43549, user_credentials={real_user=slave}} blocked reactor thread for 65513.2us
I20251025 01:55:21.192363 10959 tablet_bootstrap.cc:492] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:21.210801 10946 log.cc:1199] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/bab272a511e14adab00dd094ff1ee9e5
I20251025 01:55:21.210855 10959 ts_tablet_manager.cc:1403] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.001s
I20251025 01:55:21.211802 10946 ts_tablet_manager.cc:1950] T bab272a511e14adab00dd094ff1ee9e5 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:21.215492 10947 ts_tablet_manager.cc:1929] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 01:55:21.212049 10959 raft_consensus.cc:359] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.215750 10959 raft_consensus.cc:385] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.215863 10959 raft_consensus.cc:740] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.216152 10959 consensus_queue.cc:260] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.216662 10959 ts_tablet_manager.cc:1434] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251025 01:55:21.216984 10959 tablet_bootstrap.cc:492] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:21.218537 10959 tablet_bootstrap.cc:654] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.221858 9975 catalog_manager.cc:4985] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): tablet bab272a511e14adab00dd094ff1ee9e5 (table default.table_a2d327beb950460cbf080c4c44d89943 [id=7477ea6c70f84bc2897964f18294d49f]) successfully deleted
I20251025 01:55:21.222352 10946 tablet_replica.cc:333] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:21.222673 10946 raft_consensus.cc:2243] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:21.222915 10946 raft_consensus.cc:2272] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.224921 10946 ts_tablet_manager.cc:1916] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.224414 10957 tablet_bootstrap.cc:492] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:21.225827 10957 ts_tablet_manager.cc:1403] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.000s
I20251025 01:55:21.226830 10957 raft_consensus.cc:359] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.227062 10957 raft_consensus.cc:385] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.227190 10957 raft_consensus.cc:740] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.227708 10957 consensus_queue.cc:260] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.233299 10957 ts_tablet_manager.cc:1434] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251025 01:55:21.235963 10957 tablet_bootstrap.cc:492] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:21.238381 10957 tablet_bootstrap.cc:654] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.238452 10946 ts_tablet_manager.cc:1929] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:21.238615 10946 log.cc:1199] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/8aa21b52709149139ddb4f99d9ba228c
I20251025 01:55:21.238970 10946 ts_tablet_manager.cc:1950] T 8aa21b52709149139ddb4f99d9ba228c P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:21.239737 10947 log.cc:1199] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/9f9bd9812cc74f49b4028e89d2b5f0d7
I20251025 01:55:21.240428 10947 ts_tablet_manager.cc:1950] T 9f9bd9812cc74f49b4028e89d2b5f0d7 P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:21.243400 9975 catalog_manager.cc:4985] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]) successfully deleted
W20251025 01:55:21.245033 9975 catalog_manager.cc:4952] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): delete failed for tablet 7772240769e549ff8be322ce7ebc7ce4 because the tablet was not found. No further retry: Not found: Tablet not found: 7772240769e549ff8be322ce7ebc7ce4
I20251025 01:55:21.246914 10947 tablet_replica.cc:333] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:21.247105 10947 raft_consensus.cc:2243] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:21.247455 10947 raft_consensus.cc:2272] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.249614 10957 tablet_bootstrap.cc:492] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:21.249773 10957 ts_tablet_manager.cc:1403] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251025 01:55:21.250406 10957 raft_consensus.cc:359] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.250571 10957 raft_consensus.cc:385] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.250653 10957 raft_consensus.cc:740] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.250888 10957 consensus_queue.cc:260] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.251996 10959 tablet_bootstrap.cc:492] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:21.252139 10959 ts_tablet_manager.cc:1403] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.005s
I20251025 01:55:21.252229 10957 ts_tablet_manager.cc:1434] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 01:55:21.253190 10963 tablet_bootstrap.cc:492] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:21.254143 10963 ts_tablet_manager.cc:1403] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20251025 01:55:21.255674 10947 ts_tablet_manager.cc:1916] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.253997 10959 raft_consensus.cc:359] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.255905 10959 raft_consensus.cc:385] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.256027 10959 raft_consensus.cc:740] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.256246 10959 consensus_queue.cc:260] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.256758 10959 ts_tablet_manager.cc:1434] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 01:55:21.258224 10959 tablet_bootstrap.cc:492] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:21.260890 10959 tablet_bootstrap.cc:654] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.261020 10963 raft_consensus.cc:359] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.261217 10963 raft_consensus.cc:385] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.261314 10963 raft_consensus.cc:740] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.267453 10963 consensus_queue.cc:260] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.268128 9975 catalog_manager.cc:4985] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): tablet 8aa21b52709149139ddb4f99d9ba228c (table default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956]) successfully deleted
I20251025 01:55:21.269819 10963 ts_tablet_manager.cc:1434] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251025 01:55:21.270152 10963 tablet_bootstrap.cc:492] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:21.272027 10963 tablet_bootstrap.cc:654] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.274767 10932 mvcc.cc:204] Tried to move back new op lower bound from 7214519585726832640 to 7214519576016912384. Current Snapshot: MvccSnapshot[applied={T|T < 7214519585726832640 or (T in {7214519585726832640})}]
I20251025 01:55:21.287791 10959 tablet_bootstrap.cc:492] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:21.288106 10959 ts_tablet_manager.cc:1403] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.019s
I20251025 01:55:21.289067 10959 raft_consensus.cc:359] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.289340 10959 raft_consensus.cc:385] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.289450 10959 raft_consensus.cc:740] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.289780 10959 consensus_queue.cc:260] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.289932 10947 ts_tablet_manager.cc:1929] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:21.290074 10947 log.cc:1199] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/bab272a511e14adab00dd094ff1ee9e5
I20251025 01:55:21.293977 10959 ts_tablet_manager.cc:1434] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251025 01:55:21.299482 10947 ts_tablet_manager.cc:1950] T bab272a511e14adab00dd094ff1ee9e5 P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:21.300050 10963 tablet_bootstrap.cc:492] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:21.300225 10963 ts_tablet_manager.cc:1403] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20251025 01:55:21.300371 10947 tablet_replica.cc:333] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:21.300514 9975 catalog_manager.cc:4985] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): tablet bab272a511e14adab00dd094ff1ee9e5 (table default.table_a2d327beb950460cbf080c4c44d89943 [id=7477ea6c70f84bc2897964f18294d49f]) successfully deleted
I20251025 01:55:21.300602 10947 raft_consensus.cc:2243] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:21.300757 10947 raft_consensus.cc:2272] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.300899 10963 raft_consensus.cc:359] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.301057 10963 raft_consensus.cc:385] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.301106 10963 raft_consensus.cc:740] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.301316 10963 consensus_queue.cc:260] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.303006 10963 ts_tablet_manager.cc:1434] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 01:55:21.303390 10947 ts_tablet_manager.cc:1916] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.308152 10947 ts_tablet_manager.cc:1929] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:21.308288 10947 log.cc:1199] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/8aa21b52709149139ddb4f99d9ba228c
I20251025 01:55:21.308620 10947 ts_tablet_manager.cc:1950] T 8aa21b52709149139ddb4f99d9ba228c P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:21.309536 9975 catalog_manager.cc:4985] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): tablet 8aa21b52709149139ddb4f99d9ba228c (table default.table_083845c9d31747a68c31d8a1804a12aa [id=4ba51d95c7ed483987cb002716f0c956]) successfully deleted
I20251025 01:55:21.322803 10057 catalog_manager.cc:2184] deleted orphaned HMS catalog entry for table [id=269cfd64fbd94f0486843525917d9bb4]
I20251025 01:55:21.323151 10057 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58658 (request call id 55) took 2452 ms. Trace:
I20251025 01:55:21.323292 10057 rpcz_store.cc:276] 1025 01:55:18.870543 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:19.893213 (+1022670us) service_pool.cc:225] Handling call
1025 01:55:20.718851 (+825638us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 01:55:20.719396 (+ 545us) catalog_manager.cc:2147] Created new table and tablet info
1025 01:55:20.993267 (+273871us) catalog_manager.cc:2173] Created new table in HMS catalog
1025 01:55:20.993886 (+ 619us) write_op.cc:489] Releasing partition, row and schema locks
1025 01:55:20.993889 (+ 3us) write_op.cc:454] Released schema lock
1025 01:55:20.994086 (+ 197us) catalog_manager.cc:2181] Rolling back HMS table creation
1025 01:55:21.323114 (+329028us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 01:55:20.993832 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0ceaf0e1ce0f4c23b39a5bf4736de5ce is not leader of this config: current role FOLLOWER)
1025 01:55:20.993840 (+ 8us) write_op.cc:489] Releasing partition, row and schema locks
1025 01:55:20.993844 (+ 4us) write_op.cc:454] Released schema lock
1025 01:55:20.993847 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":347573,"Hive Metastore.run_cpu_time_us":437,"Hive Metastore.run_wall_time_us":254655,"child_traces":[["op",{}]]}
I20251025 01:55:21.361672 9983 catalog_manager.cc:2731] deleted HMS catalog entry for table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]
I20251025 01:55:21.364842 10055 catalog_manager.cc:2184] deleted orphaned HMS catalog entry for table [id=2f021b58f5304a3f8e7d46b745a7165a]
I20251025 01:55:21.365291 10055 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:58658 (request call id 56) took 2386 ms. Trace:
I20251025 01:55:21.365401 10055 rpcz_store.cc:276] 1025 01:55:18.978966 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 01:55:19.928547 (+949581us) service_pool.cc:225] Handling call
1025 01:55:20.719973 (+791426us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 01:55:20.720421 (+ 448us) catalog_manager.cc:2147] Created new table and tablet info
1025 01:55:21.128935 (+408514us) catalog_manager.cc:2173] Created new table in HMS catalog
1025 01:55:21.129625 (+ 690us) write_op.cc:489] Releasing partition, row and schema locks
1025 01:55:21.129628 (+ 3us) write_op.cc:454] Released schema lock
1025 01:55:21.129900 (+ 272us) catalog_manager.cc:2181] Rolling back HMS table creation
1025 01:55:21.365199 (+235299us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 01:55:21.129551 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0ceaf0e1ce0f4c23b39a5bf4736de5ce is not leader of this config: current role FOLLOWER)
1025 01:55:21.129564 (+ 13us) write_op.cc:489] Releasing partition, row and schema locks
1025 01:55:21.129570 (+ 6us) write_op.cc:454] Released schema lock
1025 01:55:21.129575 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":465458,"Hive Metastore.run_cpu_time_us":463,"Hive Metastore.run_wall_time_us":177442,"child_traces":[["op",{}]]}
I20251025 01:55:21.393445 9984 catalog_manager.cc:2731] deleted HMS catalog entry for table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]
I20251025 01:55:21.418115 10970 raft_consensus.cc:493] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:21.418329 10970 raft_consensus.cc:515] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.419123 10970 leader_election.cc:290] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:21.419972 10462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cceaea14d4054de09cdaacb93f9b5582" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:21.420008 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cceaea14d4054de09cdaacb93f9b5582" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:21.420225 10598 raft_consensus.cc:2468] T cceaea14d4054de09cdaacb93f9b5582 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:21.420226 10462 raft_consensus.cc:2468] T cceaea14d4054de09cdaacb93f9b5582 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:21.420682 10255 leader_election.cc:304] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:21.421440 10970 raft_consensus.cc:2804] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:21.421547 10970 raft_consensus.cc:493] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:21.421623 10970 raft_consensus.cc:3060] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:21.424384 10970 raft_consensus.cc:515] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.425195 10462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cceaea14d4054de09cdaacb93f9b5582" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:21.425380 10462 raft_consensus.cc:3060] T cceaea14d4054de09cdaacb93f9b5582 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:21.425457 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cceaea14d4054de09cdaacb93f9b5582" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:21.425638 10598 raft_consensus.cc:3060] T cceaea14d4054de09cdaacb93f9b5582 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:21.427191 10836 raft_consensus.cc:493] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:21.427372 10836 raft_consensus.cc:515] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.427940 10836 leader_election.cc:290] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:21.428027 10462 raft_consensus.cc:2468] T cceaea14d4054de09cdaacb93f9b5582 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:21.428041 10598 raft_consensus.cc:2468] T cceaea14d4054de09cdaacb93f9b5582 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:21.428573 10254 leader_election.cc:304] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:21.428570 10462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8f17943e2d0462da38dcdfb4b1cc025" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:21.428732 10462 raft_consensus.cc:2468] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:21.428746 10320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8f17943e2d0462da38dcdfb4b1cc025" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:21.428959 10320 raft_consensus.cc:2468] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:21.429070 10531 leader_election.cc:304] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2; no voters:
I20251025 01:55:21.429229 10852 raft_consensus.cc:2804] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:21.429364 10852 raft_consensus.cc:697] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:21.429605 10836 raft_consensus.cc:2804] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:21.429694 10836 raft_consensus.cc:493] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:21.429621 10852 consensus_queue.cc:237] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.429788 10836 raft_consensus.cc:3060] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:21.431140 10970 leader_election.cc:290] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:21.432614 10836 raft_consensus.cc:515] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.433537 10462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8f17943e2d0462da38dcdfb4b1cc025" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:21.433666 10462 raft_consensus.cc:3060] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:21.433952 10320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8f17943e2d0462da38dcdfb4b1cc025" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
I20251025 01:55:21.434167 10320 raft_consensus.cc:3060] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:21.436090 10462 raft_consensus.cc:2468] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:21.436434 10531 leader_election.cc:304] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2; no voters:
I20251025 01:55:21.436687 10978 raft_consensus.cc:2804] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:21.436810 10978 raft_consensus.cc:697] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Running, Role: LEADER
I20251025 01:55:21.437028 10978 consensus_queue.cc:237] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.437623 10320 raft_consensus.cc:2468] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:21.438124 10836 leader_election.cc:290] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:21.439569 9990 catalog_manager.cc:2731] deleted HMS catalog entry for table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a]
I20251025 01:55:21.458806 9982 catalog_manager.cc:2174] created HMS catalog entry for table [id=25e9ab81dd8247c2a0f8bf9913e237bc]
I20251025 01:55:21.472257 9982 catalog_manager.cc:5741] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581) does not have the latest schema for tablet f13a1ad12e2f4f2c95103cf8d76d6da6 (table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652]). Expected version 1 got 0
I20251025 01:55:21.475587 10300 tablet_service.cc:1552] Processing DeleteTablet for tablet df08f7a1b9ca4b8e8328d9295bf52576 with delete_type TABLET_DATA_DELETED (replaced by tablet 0541393def0b4dcc9cf86038e01d1c21) from {username='slave'} at 127.0.0.1:49356
W20251025 01:55:21.477015 9975 catalog_manager.cc:4952] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): delete failed for tablet df08f7a1b9ca4b8e8328d9295bf52576 because the tablet was not found. No further retry: Not found: Tablet not found: df08f7a1b9ca4b8e8328d9295bf52576
I20251025 01:55:21.479054 9981 catalog_manager.cc:2174] created HMS catalog entry for table [id=fd3f033d7c714a1d860497da7452245a]
I20251025 01:55:21.484077 10578 tablet_service.cc:1505] Processing CreateTablet for tablet 898ce3e93e56489f83a04fc46e7328de (DEFAULT_TABLE table=default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.484469 10578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898ce3e93e56489f83a04fc46e7328de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.487937 10300 tablet_service.cc:1505] Processing CreateTablet for tablet 898ce3e93e56489f83a04fc46e7328de (DEFAULT_TABLE table=default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.488401 10300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898ce3e93e56489f83a04fc46e7328de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.489674 10440 tablet_service.cc:1505] Processing CreateTablet for tablet 898ce3e93e56489f83a04fc46e7328de (DEFAULT_TABLE table=default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.490108 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898ce3e93e56489f83a04fc46e7328de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.492992 9982 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652] alter complete (version 1)
I20251025 01:55:21.495666 10959 tablet_bootstrap.cc:492] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:21.497303 10959 tablet_bootstrap.cc:654] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.510637 10959 tablet_bootstrap.cc:492] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:21.510803 10959 ts_tablet_manager.cc:1403] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20251025 01:55:21.512133 10959 raft_consensus.cc:359] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.512327 10959 raft_consensus.cc:385] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.512426 10959 raft_consensus.cc:740] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.512679 10959 consensus_queue.cc:260] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.514432 10963 tablet_bootstrap.cc:492] T 898ce3e93e56489f83a04fc46e7328de P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:21.516368 10963 tablet_bootstrap.cc:654] T 898ce3e93e56489f83a04fc46e7328de P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.516909 10959 ts_tablet_manager.cc:1434] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251025 01:55:21.516958 10957 tablet_bootstrap.cc:492] T 898ce3e93e56489f83a04fc46e7328de P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:21.517901 9989 catalog_manager.cc:2174] created HMS catalog entry for table [id=5583c1b10fd14168a2bd689a0bd5e433]
I20251025 01:55:21.518510 10957 tablet_bootstrap.cc:654] T 898ce3e93e56489f83a04fc46e7328de P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.541232 10963 tablet_bootstrap.cc:492] T 898ce3e93e56489f83a04fc46e7328de P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:21.541399 10963 ts_tablet_manager.cc:1403] T 898ce3e93e56489f83a04fc46e7328de P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.015s
I20251025 01:55:21.541392 9982 catalog_manager.cc:5741] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581) does not have the latest schema for tablet b8f17943e2d0462da38dcdfb4b1cc025 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]). Expected version 1 got 0
I20251025 01:55:21.541596 9982 catalog_manager.cc:5741] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581) does not have the latest schema for tablet 0541393def0b4dcc9cf86038e01d1c21 (table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c]). Expected version 1 got 0
I20251025 01:55:21.541759 9982 catalog_manager.cc:5649] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b reported cstate change: term changed from 0 to 1, leader changed from <none> to c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129). New cstate: current_term: 1 leader_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20251025 01:55:21.542202 9982 catalog_manager.cc:5741] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581) does not have the latest schema for tablet cceaea14d4054de09cdaacb93f9b5582 (table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f]). Expected version 1 got 0
I20251025 01:55:21.542173 10963 raft_consensus.cc:359] T 898ce3e93e56489f83a04fc46e7328de P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.542438 10963 raft_consensus.cc:385] T 898ce3e93e56489f83a04fc46e7328de P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.542546 10963 raft_consensus.cc:740] T 898ce3e93e56489f83a04fc46e7328de P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.542753 10963 consensus_queue.cc:260] T 898ce3e93e56489f83a04fc46e7328de P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.546269 10957 tablet_bootstrap.cc:492] T 898ce3e93e56489f83a04fc46e7328de P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:21.547031 9987 catalog_manager.cc:2174] created HMS catalog entry for table [id=f8c139c823a940e49fe874c9e4035ca0]
I20251025 01:55:21.547304 10957 ts_tablet_manager.cc:1403] T 898ce3e93e56489f83a04fc46e7328de P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.011s
I20251025 01:55:21.549780 10963 ts_tablet_manager.cc:1434] T 898ce3e93e56489f83a04fc46e7328de P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251025 01:55:21.551982 9981 master_service.cc:954] ReplaceTablet: received request to replace tablet 9f9bd9812cc74f49b4028e89d2b5f0d7 from {username='slave'} at 127.0.0.1:40888
I20251025 01:55:21.553030 9981 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_c18eb2691b2a4c5daf9746ee47a5e451" } new_table_name: "default.Table_7bca7380f82c44a4ac29fb6f6ed8a2fb" modify_external_catalogs: true
I20251025 01:55:21.559276 10957 raft_consensus.cc:359] T 898ce3e93e56489f83a04fc46e7328de P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.559666 10957 raft_consensus.cc:385] T 898ce3e93e56489f83a04fc46e7328de P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.559834 10957 raft_consensus.cc:740] T 898ce3e93e56489f83a04fc46e7328de P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.560173 10957 consensus_queue.cc:260] T 898ce3e93e56489f83a04fc46e7328de P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.562587 10957 ts_tablet_manager.cc:1434] T 898ce3e93e56489f83a04fc46e7328de P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20251025 01:55:21.569116 9989 catalog_manager.cc:5741] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021) does not have the latest schema for tablet 0541393def0b4dcc9cf86038e01d1c21 (table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c]). Expected version 1 got 0
I20251025 01:55:21.569269 10300 tablet_service.cc:1505] Processing CreateTablet for tablet a1269807c3644c629ed5d0e4e45c5efc (DEFAULT_TABLE table=default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.569392 9989 catalog_manager.cc:5741] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021) does not have the latest schema for tablet cceaea14d4054de09cdaacb93f9b5582 (table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f]). Expected version 1 got 0
I20251025 01:55:21.569737 10300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1269807c3644c629ed5d0e4e45c5efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.569602 9989 catalog_manager.cc:5649] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131). New cstate: current_term: 1 leader_uuid: "90109d2fdb464942b20891b5856f8ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: HEALTHY } } }
I20251025 01:55:21.569973 9989 catalog_manager.cc:5741] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021) does not have the latest schema for tablet b8f17943e2d0462da38dcdfb4b1cc025 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]). Expected version 1 got 0
I20251025 01:55:21.572746 9982 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811] alter complete (version 1)
W20251025 01:55:21.573650 9982 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet b8f17943e2d0462da38dcdfb4b1cc025 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet b8f17943e2d0462da38dcdfb4b1cc025 consensus state has no leader
I20251025 01:55:21.576371 10578 tablet_service.cc:1505] Processing CreateTablet for tablet a1269807c3644c629ed5d0e4e45c5efc (DEFAULT_TABLE table=default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.576988 10440 tablet_service.cc:1505] Processing CreateTablet for tablet a1269807c3644c629ed5d0e4e45c5efc (DEFAULT_TABLE table=default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.580144 10578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1269807c3644c629ed5d0e4e45c5efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.582479 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1269807c3644c629ed5d0e4e45c5efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.584471 10959 tablet_bootstrap.cc:492] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:21.586050 10959 tablet_bootstrap.cc:654] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.593113 10957 tablet_bootstrap.cc:492] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:21.593593 9985 catalog_manager.cc:2174] created HMS catalog entry for table [id=35ac4045482e4025b49f122add8f0889]
I20251025 01:55:21.595026 10957 tablet_bootstrap.cc:654] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.598812 9982 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_827ca3c8c36543b29dac05ab520c552a" } new_table_name: "default.Table_6670e4ea42324ddc9e0c5a4317baf960" modify_external_catalogs: true
I20251025 01:55:21.600617 10957 tablet_bootstrap.cc:492] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:21.600773 10957 ts_tablet_manager.cc:1403] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20251025 01:55:21.601491 10957 raft_consensus.cc:359] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.601740 10957 raft_consensus.cc:385] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.601811 10957 raft_consensus.cc:740] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.602162 10957 consensus_queue.cc:260] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.605533 10957 ts_tablet_manager.cc:1434] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251025 01:55:21.609222 10963 tablet_bootstrap.cc:492] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:21.610904 10963 tablet_bootstrap.cc:654] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.613374 10959 tablet_bootstrap.cc:492] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:21.613431 9989 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_328c63794848430fb0f74a446d210d94" } new_table_name: "default.Table_68a9e5b0ffdd4bd081b95dca5aff0202" modify_external_catalogs: true
I20251025 01:55:21.613518 10959 ts_tablet_manager.cc:1403] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.005s
I20251025 01:55:21.614490 10959 raft_consensus.cc:359] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.614874 9985 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_ac99b534589f45e6871ddfa3a1331b16" } new_table_name: "default.Table_5180205af52d470586f94f073f3537ca" modify_external_catalogs: true
I20251025 01:55:21.615939 10959 raft_consensus.cc:385] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.616037 10959 raft_consensus.cc:740] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.616274 10959 consensus_queue.cc:260] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.616799 10959 ts_tablet_manager.cc:1434] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 01:55:21.630234 10300 tablet_service.cc:1505] Processing CreateTablet for tablet 3d37511aec3746cea09c1ba224cb14db (DEFAULT_TABLE table=default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.630374 9987 catalog_manager.cc:5741] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549) does not have the latest schema for tablet cceaea14d4054de09cdaacb93f9b5582 (table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f]). Expected version 1 got 0
I20251025 01:55:21.630604 9987 catalog_manager.cc:5741] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549) does not have the latest schema for tablet 0541393def0b4dcc9cf86038e01d1c21 (table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c]). Expected version 1 got 0
I20251025 01:55:21.630645 10300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d37511aec3746cea09c1ba224cb14db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.630815 9987 catalog_manager.cc:5741] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549) does not have the latest schema for tablet b8f17943e2d0462da38dcdfb4b1cc025 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]). Expected version 1 got 0
I20251025 01:55:21.634723 9988 catalog_manager.cc:2174] created HMS catalog entry for table [id=46a03684abfd437997c4881af2f5b980]
I20251025 01:55:21.637720 10576 tablet_service.cc:1505] Processing CreateTablet for tablet 3d37511aec3746cea09c1ba224cb14db (DEFAULT_TABLE table=default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.638276 10576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d37511aec3746cea09c1ba224cb14db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.639120 10301 tablet_service.cc:1505] Processing CreateTablet for tablet 14fdd89953bc49fb91adda5d2a840f3a (DEFAULT_TABLE table=default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.639577 10301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fdd89953bc49fb91adda5d2a840f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.648336 10576 tablet_service.cc:1505] Processing CreateTablet for tablet 14fdd89953bc49fb91adda5d2a840f3a (DEFAULT_TABLE table=default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.648902 10576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fdd89953bc49fb91adda5d2a840f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.649410 9988 master_service.cc:954] ReplaceTablet: received request to replace tablet db0c31aa1beb45b0936620975739ed18 from {username='slave'} at 127.0.0.1:40888
I20251025 01:55:21.651114 10959 tablet_bootstrap.cc:492] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:21.652954 9987 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_986ff2b14daa4beebda43706d47dc98b" } modify_external_catalogs: true
I20251025 01:55:21.653138 10959 tablet_bootstrap.cc:654] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.653201 9987 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_986ff2b14daa4beebda43706d47dc98b" } modify_external_catalogs: true
I20251025 01:55:21.654552 10957 tablet_bootstrap.cc:492] T 3d37511aec3746cea09c1ba224cb14db P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:21.658985 9988 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Sending DeleteTablet for 3 replicas of tablet db0c31aa1beb45b0936620975739ed18
I20251025 01:55:21.659448 9988 catalog_manager.cc:6719] ReplaceTablet: tablet db0c31aa1beb45b0936620975739ed18 deleted and replaced by tablet 5a2f9794447f4d7e8f6f548e4b26c8c6
I20251025 01:55:21.659700 9988 master_service.cc:954] ReplaceTablet: received request to replace tablet df08f7a1b9ca4b8e8328d9295bf52576 from {username='slave'} at 127.0.0.1:40888
I20251025 01:55:21.660324 9988 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_8553cdea51804bfa861b6c10537b99c3" } new_table_name: "default.Table_89a70ae51ea04c43b2f1842ec3f87253" modify_external_catalogs: true
I20251025 01:55:21.662796 10576 tablet_service.cc:1552] Processing DeleteTablet for tablet db0c31aa1beb45b0936620975739ed18 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a2f9794447f4d7e8f6f548e4b26c8c6) from {username='slave'} at 127.0.0.1:43660
I20251025 01:55:21.663094 10947 tablet_replica.cc:333] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:21.663324 10947 raft_consensus.cc:2243] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:21.663553 10947 raft_consensus.cc:2272] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.664202 10947 ts_tablet_manager.cc:1916] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.666169 10301 tablet_service.cc:1552] Processing DeleteTablet for tablet db0c31aa1beb45b0936620975739ed18 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a2f9794447f4d7e8f6f548e4b26c8c6) from {username='slave'} at 127.0.0.1:49356
I20251025 01:55:21.666466 10946 tablet_replica.cc:333] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:21.666682 10946 raft_consensus.cc:2243] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:21.666885 10946 raft_consensus.cc:2272] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.667559 10440 tablet_service.cc:1505] Processing CreateTablet for tablet 3d37511aec3746cea09c1ba224cb14db (DEFAULT_TABLE table=default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.667725 10946 ts_tablet_manager.cc:1916] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.668053 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d37511aec3746cea09c1ba224cb14db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.671541 10947 ts_tablet_manager.cc:1929] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:21.671696 10947 log.cc:1199] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/db0c31aa1beb45b0936620975739ed18
I20251025 01:55:21.672433 10441 tablet_service.cc:1505] Processing CreateTablet for tablet 14fdd89953bc49fb91adda5d2a840f3a (DEFAULT_TABLE table=default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.672789 10947 ts_tablet_manager.cc:1950] T db0c31aa1beb45b0936620975739ed18 P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:21.672817 10441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fdd89953bc49fb91adda5d2a840f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.673803 9975 catalog_manager.cc:4985] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): tablet db0c31aa1beb45b0936620975739ed18 (table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]) successfully deleted
I20251025 01:55:21.675400 10439 tablet_service.cc:1552] Processing DeleteTablet for tablet db0c31aa1beb45b0936620975739ed18 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a2f9794447f4d7e8f6f548e4b26c8c6) from {username='slave'} at 127.0.0.1:58374
I20251025 01:55:21.676432 10949 tablet_replica.cc:333] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:21.676501 10957 tablet_bootstrap.cc:654] T 3d37511aec3746cea09c1ba224cb14db P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.676661 10949 raft_consensus.cc:2243] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:21.677013 10949 raft_consensus.cc:2272] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.677575 10949 ts_tablet_manager.cc:1916] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.684568 10462 raft_consensus.cc:1275] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Refusing update from remote peer 90109d2fdb464942b20891b5856f8ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 01:55:21.684621 9986 catalog_manager.cc:2174] created HMS catalog entry for table [id=c799aad2d817457cb9d21865a082a737]
I20251025 01:55:21.685140 10978 consensus_queue.cc:1048] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:21.693176 9986 master_service.cc:954] ReplaceTablet: received request to replace tablet b8f17943e2d0462da38dcdfb4b1cc025 from {username='slave'} at 127.0.0.1:40888
F20251025 01:55:21.696471 10204 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 4 out of 13 received events: NotificationEvent(eventId=24, eventTime=1761357321, eventType=CREATE_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_a981c91dba574e1c8ccce55a41c4bd88, message={"server":"","servicePrincipal":"","db":"default","table":"table_a981c91dba574e1c8ccce55a41c4bd88","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a981c91dba574e1c8ccce55a41c4bd88\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357320},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_a981c91dba574e1c8ccce55a41c4bd88\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a981c91dba574e1c8ccce55a41c4bd88\",\"kudu.table_id\":\"269cfd64fbd94f0486843525917d9bb4\",\"transient_lastDdlTime\":\"1761357320\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_44e4ccc4f2de47198a0599e2577108cf, message={"server":"","servicePrincipal":"","db":"default","table":"table_44e4ccc4f2de47198a0599e2577108cf","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_44e4ccc4f2de47198a0599e2577108cf\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357319},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_44e4ccc4f2de47198a0599e2577108cf\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_44e4ccc4f2de47198a0599e2577108cf\",\"kudu.table_id\":\"58afab4104e84c868eeccda411b8452b\",\"transient_lastDdlTime\":\"1761357319\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251025 01:55:21.702955 10320 raft_consensus.cc:1275] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Refusing update from remote peer 90109d2fdb464942b20891b5856f8ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 01:55:21.703991 10949 ts_tablet_manager.cc:1929] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:21.704493 10978 consensus_queue.cc:1048] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:21.704807 10949 log.cc:1199] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/db0c31aa1beb45b0936620975739ed18
I20251025 01:55:21.705356 10949 ts_tablet_manager.cc:1950] T db0c31aa1beb45b0936620975739ed18 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:21.708500 10959 tablet_bootstrap.cc:492] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
@ 0x7f35355fedcd google::LogMessage::Fail() at ??:0
I20251025 01:55:21.708678 10959 ts_tablet_manager.cc:1403] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.058s user 0.016s sys 0.011s
I20251025 01:55:21.709632 10959 raft_consensus.cc:359] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.709848 10959 raft_consensus.cc:385] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.709940 10959 raft_consensus.cc:740] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
@ 0x7f3535602b93 google::LogMessage::SendToLog() at ??:0
I20251025 01:55:21.710182 10959 consensus_queue.cc:260] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.710793 10959 ts_tablet_manager.cc:1434] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 01:55:21.711108 10959 tablet_bootstrap.cc:492] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:21.712859 10959 tablet_bootstrap.cc:654] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.713711 10852 raft_consensus.cc:493] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:21.713831 9986 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Sending DeleteTablet for 3 replicas of tablet b8f17943e2d0462da38dcdfb4b1cc025
I20251025 01:55:21.713883 10852 raft_consensus.cc:515] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.714512 10852 leader_election.cc:290] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:21.714406 9986 catalog_manager.cc:6719] ReplaceTablet: tablet b8f17943e2d0462da38dcdfb4b1cc025 deleted and replaced by tablet d7df9d477cf1448d91cd8cac0ab321a9
I20251025 01:55:21.716023 10578 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f17943e2d0462da38dcdfb4b1cc025 with delete_type TABLET_DATA_DELETED (replaced by tablet d7df9d477cf1448d91cd8cac0ab321a9) from {username='slave'} at 127.0.0.1:43660
I20251025 01:55:21.716902 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1269807c3644c629ed5d0e4e45c5efc" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:21.717144 10598 raft_consensus.cc:2468] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
@ 0x7f35355fe7cc google::LogMessage::Flush() at ??:0
I20251025 01:55:21.717700 10255 leader_election.cc:304] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:21.718247 10299 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f17943e2d0462da38dcdfb4b1cc025 with delete_type TABLET_DATA_DELETED (replaced by tablet d7df9d477cf1448d91cd8cac0ab321a9) from {username='slave'} at 127.0.0.1:49356
I20251025 01:55:21.718247 10852 raft_consensus.cc:2804] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:21.718472 10852 raft_consensus.cc:493] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:21.718564 10852 raft_consensus.cc:3060] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
@ 0x7f35355fea11 google::LogMessage::~LogMessage() at ??:0
I20251025 01:55:21.725901 9986 master_service.cc:954] ReplaceTablet: received request to replace tablet 95491aa8141440fc999a3c742a24d503 from {username='slave'} at 127.0.0.1:40888
I20251025 01:55:21.726763 10946 ts_tablet_manager.cc:1929] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:21.730297 10301 tablet_service.cc:1505] Processing CreateTablet for tablet 5906028cad8944bb97e90be8be25003f (DEFAULT_TABLE table=default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.730715 10301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5906028cad8944bb97e90be8be25003f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.735152 9986 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Sending DeleteTablet for 3 replicas of tablet 95491aa8141440fc999a3c742a24d503
I20251025 01:55:21.736204 10578 tablet_service.cc:1552] Processing DeleteTablet for tablet 95491aa8141440fc999a3c742a24d503 with delete_type TABLET_DATA_DELETED (replaced by tablet 33df938c44154f41b333523a51b65e4e) from {username='slave'} at 127.0.0.1:43660
I20251025 01:55:21.736279 9986 catalog_manager.cc:6719] ReplaceTablet: tablet 95491aa8141440fc999a3c742a24d503 deleted and replaced by tablet 33df938c44154f41b333523a51b65e4e
I20251025 01:55:21.736598 9986 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_9187de27a70f48e08cc7284131fe543d" } modify_external_catalogs: true
I20251025 01:55:21.736690 10299 tablet_service.cc:1552] Processing DeleteTablet for tablet 95491aa8141440fc999a3c742a24d503 with delete_type TABLET_DATA_DELETED (replaced by tablet 33df938c44154f41b333523a51b65e4e) from {username='slave'} at 127.0.0.1:49356
I20251025 01:55:21.736819 9986 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "default.Table_9187de27a70f48e08cc7284131fe543d" } modify_external_catalogs: true
I20251025 01:55:21.737337 10947 tablet_replica.cc:333] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:21.739964 10852 raft_consensus.cc:515] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.740607 10852 leader_election.cc:290] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
@ 0x7f354bfb7b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251025 01:55:21.742175 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1269807c3644c629ed5d0e4e45c5efc" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:21.742358 10598 raft_consensus.cc:3060] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:21.745694 10598 raft_consensus.cc:2468] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:21.746186 10255 leader_election.cc:304] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:21.746433 10852 raft_consensus.cc:2804] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:21.746641 10852 raft_consensus.cc:697] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:21.746922 10852 consensus_queue.cc:237] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.726887 10946 log.cc:1199] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/db0c31aa1beb45b0936620975739ed18
I20251025 01:55:21.747977 10946 ts_tablet_manager.cc:1950] T db0c31aa1beb45b0936620975739ed18 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:21.748926 9975 catalog_manager.cc:4985] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): tablet db0c31aa1beb45b0936620975739ed18 (table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]) successfully deleted
I20251025 01:55:21.749328 10946 tablet_replica.cc:333] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:21.717042 10460 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1269807c3644c629ed5d0e4e45c5efc" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
W20251025 01:55:21.750269 10254 leader_election.cc:343] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Illegal state: must be running to vote when last-logged opid is not known
I20251025 01:55:21.751062 9974 catalog_manager.cc:4985] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): tablet db0c31aa1beb45b0936620975739ed18 (table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]) successfully deleted
I20251025 01:55:21.729358 10441 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f17943e2d0462da38dcdfb4b1cc025 with delete_type TABLET_DATA_DELETED (replaced by tablet d7df9d477cf1448d91cd8cac0ab321a9) from {username='slave'} at 127.0.0.1:58374
I20251025 01:55:21.736116 10440 tablet_service.cc:1552] Processing DeleteTablet for tablet 95491aa8141440fc999a3c742a24d503 with delete_type TABLET_DATA_DELETED (replaced by tablet 33df938c44154f41b333523a51b65e4e) from {username='slave'} at 127.0.0.1:58374
I20251025 01:55:21.741109 10439 tablet_service.cc:1505] Processing CreateTablet for tablet 5906028cad8944bb97e90be8be25003f (DEFAULT_TABLE table=default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.742178 10459 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1269807c3644c629ed5d0e4e45c5efc" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:21.751696 10439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5906028cad8944bb97e90be8be25003f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:21.752985 10959 tablet_bootstrap.cc:492] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:21.753154 10959 ts_tablet_manager.cc:1403] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.042s user 0.020s sys 0.016s
I20251025 01:55:21.754011 10959 raft_consensus.cc:359] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.754259 10959 raft_consensus.cc:385] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.754336 10959 raft_consensus.cc:740] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.754626 10959 consensus_queue.cc:260] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.755355 10959 ts_tablet_manager.cc:1434] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
@ 0x7f354bfb6411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
W20251025 01:55:21.756536 10254 leader_election.cc:343] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Illegal state: must be running to vote when last-logged opid is not known
@ 0x7f354bfbe08c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251025 01:55:21.761435 10963 tablet_bootstrap.cc:492] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:21.761611 10963 ts_tablet_manager.cc:1403] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.152s user 0.005s sys 0.003s
I20251025 01:55:21.761605 10578 tablet_service.cc:1505] Processing CreateTablet for tablet 5906028cad8944bb97e90be8be25003f (DEFAULT_TABLE table=default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.762045 10578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5906028cad8944bb97e90be8be25003f. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f354bfbdf09 std::_Function_handler<>::_M_invoke() at ??:0
I20251025 01:55:21.762462 10963 raft_consensus.cc:359] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.762692 10963 raft_consensus.cc:385] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.762769 10963 raft_consensus.cc:740] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.763034 10963 consensus_queue.cc:260] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.763620 10963 ts_tablet_manager.cc:1434] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 01:55:21.763929 10963 tablet_bootstrap.cc:492] T 3d37511aec3746cea09c1ba224cb14db P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:21.765901 10963 tablet_bootstrap.cc:654] T 3d37511aec3746cea09c1ba224cb14db P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.767997 11011 tablet.cc:1722] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
@ 0xa8867d std::function<>::operator()() at ??:?
I20251025 01:55:21.774266 10299 tablet_service.cc:1505] Processing CreateTablet for tablet 185770378a5b459d9bd420c7fe96c7e1 (DEFAULT_TABLE table=default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.774682 10299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 185770378a5b459d9bd420c7fe96c7e1. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f3538372ee9 kudu::Thread::SuperviseThread() at ??:0
I20251025 01:55:21.781030 10959 tablet_bootstrap.cc:492] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:21.782109 10949 tablet_replica.cc:333] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:21.782352 10949 raft_consensus.cc:2243] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus shutting down.
@ 0x7f353b97a6db start_thread at ??:0
I20251025 01:55:21.782691 10949 raft_consensus.cc:2272] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.782872 10959 tablet_bootstrap.cc:654] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.784129 10441 tablet_service.cc:1505] Processing CreateTablet for tablet 185770378a5b459d9bd420c7fe96c7e1 (DEFAULT_TABLE table=default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:21.784497 10441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 185770378a5b459d9bd420c7fe96c7e1. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f35344af71f clone at ??:0
I20251025 01:55:21.790213 10949 ts_tablet_manager.cc:1916] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.797323 11010 tablet.cc:1722] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 01:55:21.797986 10598 raft_consensus.cc:1275] T cceaea14d4054de09cdaacb93f9b5582 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Refusing update from remote peer c3e4f9f2253e416e8b31ea86ead23a3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 01:55:21.798835 10970 consensus_queue.cc:1048] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:21.799861 10959 tablet_bootstrap.cc:492] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:21.799983 10959 ts_tablet_manager.cc:1403] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.003s
I20251025 01:55:21.800799 10959 raft_consensus.cc:359] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.800961 10959 raft_consensus.cc:385] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.801040 10959 raft_consensus.cc:740] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.801246 10959 consensus_queue.cc:260] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.801755 10959 ts_tablet_manager.cc:1434] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 01:55:21.804277 10957 tablet_bootstrap.cc:492] T 3d37511aec3746cea09c1ba224cb14db P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:21.804457 10957 ts_tablet_manager.cc:1403] T 3d37511aec3746cea09c1ba224cb14db P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.150s user 0.012s sys 0.035s
I20251025 01:55:21.807193 10949 ts_tablet_manager.cc:1929] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:21.807332 10949 log.cc:1199] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/95491aa8141440fc999a3c742a24d503
I20251025 01:55:21.807701 10949 ts_tablet_manager.cc:1950] T 95491aa8141440fc999a3c742a24d503 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:21.807729 10947 raft_consensus.cc:2243] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:21.808939 10959 tablet_bootstrap.cc:492] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:21.809852 10947 raft_consensus.cc:2272] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.811558 10947 ts_tablet_manager.cc:1916] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.812232 10461 raft_consensus.cc:1275] T cceaea14d4054de09cdaacb93f9b5582 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Refusing update from remote peer c3e4f9f2253e416e8b31ea86ead23a3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 01:55:21.813225 10970 consensus_queue.cc:1048] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:21.819303 10959 tablet_bootstrap.cc:654] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.821198 11010 tablet.cc:1722] T cceaea14d4054de09cdaacb93f9b5582 P 90109d2fdb464942b20891b5856f8ac2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 01:55:21.823221 11017 tablet.cc:1722] T cceaea14d4054de09cdaacb93f9b5582 P c3e4f9f2253e416e8b31ea86ead23a3b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 01:55:21.831682 10947 ts_tablet_manager.cc:1929] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 01:55:21.831840 10947 log.cc:1199] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/b8f17943e2d0462da38dcdfb4b1cc025
I20251025 01:55:21.832250 10947 ts_tablet_manager.cc:1950] T b8f17943e2d0462da38dcdfb4b1cc025 P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:21.833451 10947 tablet_replica.cc:333] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:21.833623 10947 raft_consensus.cc:2243] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:21.833837 10947 raft_consensus.cc:2272] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.834781 11006 mvcc.cc:204] Tried to move back new op lower bound from 7214519590077665280 to 7214519588577841152. Current Snapshot: MvccSnapshot[applied={T|T < 7214519590077665280}]
I20251025 01:55:21.835021 10957 raft_consensus.cc:359] T 3d37511aec3746cea09c1ba224cb14db P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.835266 10957 raft_consensus.cc:385] T 3d37511aec3746cea09c1ba224cb14db P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.835407 10957 raft_consensus.cc:740] T 3d37511aec3746cea09c1ba224cb14db P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.836184 10957 consensus_queue.cc:260] T 3d37511aec3746cea09c1ba224cb14db P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.837129 10957 ts_tablet_manager.cc:1434] T 3d37511aec3746cea09c1ba224cb14db P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.032s user 0.002s sys 0.001s
I20251025 01:55:21.841162 10946 raft_consensus.cc:2243] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:21.841647 10946 pending_rounds.cc:70] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b: Trying to abort 2 pending ops.
I20251025 01:55:21.841856 10957 tablet_bootstrap.cc:492] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:21.841849 10946 pending_rounds.cc:77] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7214519589613035520 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "b8f17943e2d0462da38dcdfb4b1cc025" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.table_401d389205a14ac191081b5d0df151e6" dest_uuid: "90109d2fdb464942b20891b5856f8ac2" new_extra_config { } }
I20251025 01:55:21.842244 10946 pending_rounds.cc:77] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214519588607823872 op_type: NO_OP noop_request { }
I20251025 01:55:21.842366 10946 raft_consensus.cc:2889] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251025 01:55:21.842463 10946 raft_consensus.cc:2272] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.842872 11011 tablet.cc:1722] T cceaea14d4054de09cdaacb93f9b5582 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 01:55:21.844502 10957 tablet_bootstrap.cc:654] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
W20251025 01:55:21.841648 11016 consensus_queue.cc:1175] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20251025 01:55:21.845767 10946 ts_tablet_manager.cc:1916] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.853773 10946 ts_tablet_manager.cc:1929] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 01:55:21.853906 10946 log.cc:1199] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/b8f17943e2d0462da38dcdfb4b1cc025
I20251025 01:55:21.854249 10946 ts_tablet_manager.cc:1950] T b8f17943e2d0462da38dcdfb4b1cc025 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:21.856220 10946 tablet_replica.cc:333] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:21.856365 10957 tablet_bootstrap.cc:492] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:21.856376 10946 raft_consensus.cc:2243] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:21.856506 10957 ts_tablet_manager.cc:1403] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.002s
I20251025 01:55:21.856830 10946 raft_consensus.cc:2272] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.857175 10957 raft_consensus.cc:359] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.857412 10957 raft_consensus.cc:385] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.857477 10957 raft_consensus.cc:740] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.857529 10947 ts_tablet_manager.cc:1916] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.857709 10957 consensus_queue.cc:260] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.860134 10957 ts_tablet_manager.cc:1434] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 01:55:21.863934 10957 tablet_bootstrap.cc:492] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:21.868026 10957 tablet_bootstrap.cc:654] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.872531 10946 ts_tablet_manager.cc:1916] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.873525 10949 tablet_replica.cc:333] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:21.873574 10978 raft_consensus.cc:493] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:21.873716 10949 raft_consensus.cc:2243] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:21.873708 10978 raft_consensus.cc:515] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.873924 10949 raft_consensus.cc:2272] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:21.874261 10978 leader_election.cc:290] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:21.875021 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0541393def0b4dcc9cf86038e01d1c21" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:21.875195 10461 raft_consensus.cc:2468] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:21.875631 10531 leader_election.cc:304] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2; no voters:
I20251025 01:55:21.875893 10978 raft_consensus.cc:2804] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:21.875977 10320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0541393def0b4dcc9cf86038e01d1c21" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:21.876096 10963 tablet_bootstrap.cc:492] T 3d37511aec3746cea09c1ba224cb14db P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:21.876138 10978 raft_consensus.cc:493] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:21.876153 10320 raft_consensus.cc:2468] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:21.876219 10963 ts_tablet_manager.cc:1403] T 3d37511aec3746cea09c1ba224cb14db P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.011s
I20251025 01:55:21.876241 10978 raft_consensus.cc:3060] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:21.877017 10949 ts_tablet_manager.cc:1916] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:21.877017 10963 raft_consensus.cc:359] T 3d37511aec3746cea09c1ba224cb14db P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.877205 10963 raft_consensus.cc:385] T 3d37511aec3746cea09c1ba224cb14db P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.877285 10963 raft_consensus.cc:740] T 3d37511aec3746cea09c1ba224cb14db P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.877504 10963 consensus_queue.cc:260] T 3d37511aec3746cea09c1ba224cb14db P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.877961 10963 ts_tablet_manager.cc:1434] T 3d37511aec3746cea09c1ba224cb14db P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 01:55:21.878333 10963 tablet_bootstrap.cc:492] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:21.878854 10978 raft_consensus.cc:515] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.879969 10963 tablet_bootstrap.cc:654] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.880792 10977 raft_consensus.cc:493] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:21.880913 10977 raft_consensus.cc:515] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.881525 10977 leader_election.cc:290] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:21.882699 10978 leader_election.cc:290] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:21.882665 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14fdd89953bc49fb91adda5d2a840f3a" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:21.883020 10321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0541393def0b4dcc9cf86038e01d1c21" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
I20251025 01:55:21.883066 10320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14fdd89953bc49fb91adda5d2a840f3a" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:21.883212 10321 raft_consensus.cc:3060] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:21.883244 10320 raft_consensus.cc:2468] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:21.883612 10532 leader_election.cc:304] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:21.883838 10978 raft_consensus.cc:2804] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:21.883927 10978 raft_consensus.cc:493] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:21.884007 10978 raft_consensus.cc:3060] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:21.885771 10321 raft_consensus.cc:2468] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:21.886314 10532 leader_election.cc:304] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:21.886475 10978 raft_consensus.cc:515] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.886799 10947 ts_tablet_manager.cc:1929] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:21.886916 10947 log.cc:1199] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/95491aa8141440fc999a3c742a24d503
W20251025 01:55:21.887023 10531 leader_election.cc:343] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Illegal state: must be running to vote when last-logged opid is not known
I20251025 01:55:21.887501 10947 ts_tablet_manager.cc:1950] T 95491aa8141440fc999a3c742a24d503 P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:21.887609 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0541393def0b4dcc9cf86038e01d1c21" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:21.887770 10461 raft_consensus.cc:3060] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:21.889521 10946 ts_tablet_manager.cc:1929] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:21.889670 10946 log.cc:1199] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/95491aa8141440fc999a3c742a24d503
I20251025 01:55:21.890084 10946 ts_tablet_manager.cc:1950] T 95491aa8141440fc999a3c742a24d503 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:21.890348 10461 raft_consensus.cc:2468] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:21.890631 10977 raft_consensus.cc:2804] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:21.890789 10977 raft_consensus.cc:697] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Running, Role: LEADER
I20251025 01:55:21.890844 10957 tablet_bootstrap.cc:492] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:21.891050 10977 consensus_queue.cc:237] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.892751 10963 tablet_bootstrap.cc:492] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:21.892899 10963 ts_tablet_manager.cc:1403] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20251025 01:55:21.893013 10320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14fdd89953bc49fb91adda5d2a840f3a" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
I20251025 01:55:21.893200 10320 raft_consensus.cc:3060] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:21.893179 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14fdd89953bc49fb91adda5d2a840f3a" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:21.893615 10963 raft_consensus.cc:359] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
W20251025 01:55:21.893740 10531 leader_election.cc:343] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Illegal state: must be running to vote when last-logged opid is not known
I20251025 01:55:21.893791 10963 raft_consensus.cc:385] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.893846 10963 raft_consensus.cc:740] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.894140 10963 consensus_queue.cc:260] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.894619 10963 ts_tablet_manager.cc:1434] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 01:55:21.894970 10963 tablet_bootstrap.cc:492] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:21.895987 10320 raft_consensus.cc:2468] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:21.896330 10532 leader_election.cc:304] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c
I20251025 01:55:21.896510 10963 tablet_bootstrap.cc:654] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.896613 10836 raft_consensus.cc:2804] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:21.891057 10957 ts_tablet_manager.cc:1403] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.003s
I20251025 01:55:21.887081 10978 leader_election.cc:290] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Requested vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:21.901067 10957 raft_consensus.cc:359] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.901221 10957 raft_consensus.cc:385] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.901290 10957 raft_consensus.cc:740] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.901476 10957 consensus_queue.cc:260] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.901938 10957 ts_tablet_manager.cc:1434] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 01:55:21.904628 10959 tablet_bootstrap.cc:492] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:21.904786 10959 ts_tablet_manager.cc:1403] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.004s
I20251025 01:55:21.905514 10959 raft_consensus.cc:359] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.905705 10959 raft_consensus.cc:385] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.905782 10959 raft_consensus.cc:740] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.905982 10959 consensus_queue.cc:260] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.906420 10959 ts_tablet_manager.cc:1434] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 01:55:21.915674 10949 ts_tablet_manager.cc:1929] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 01:55:21.915777 10949 log.cc:1199] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/b8f17943e2d0462da38dcdfb4b1cc025
I20251025 01:55:21.930094 11018 raft_consensus.cc:493] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:21.934067 11018 raft_consensus.cc:515] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.916131 10949 ts_tablet_manager.cc:1950] T b8f17943e2d0462da38dcdfb4b1cc025 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:21.934799 11018 leader_election.cc:290] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:21.924631 10963 tablet_bootstrap.cc:492] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:21.935232 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d37511aec3746cea09c1ba224cb14db" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:21.935484 10963 ts_tablet_manager.cc:1403] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.010s
I20251025 01:55:21.935494 10461 raft_consensus.cc:2468] T 3d37511aec3746cea09c1ba224cb14db P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:21.936077 10254 leader_election.cc:304] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:21.936359 11018 raft_consensus.cc:2804] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:21.936275 10963 raft_consensus.cc:359] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.936455 10963 raft_consensus.cc:385] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.936457 11018 raft_consensus.cc:493] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:21.936530 10963 raft_consensus.cc:740] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.936540 11018 raft_consensus.cc:3060] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:21.936764 10963 consensus_queue.cc:260] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.923620 10836 raft_consensus.cc:697] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Running, Role: LEADER
I20251025 01:55:21.948948 10963 ts_tablet_manager.cc:1434] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20251025 01:55:21.948905 10836 consensus_queue.cc:237] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.949234 10963 tablet_bootstrap.cc:492] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:21.935482 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d37511aec3746cea09c1ba224cb14db" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:21.950023 10596 raft_consensus.cc:2468] T 3d37511aec3746cea09c1ba224cb14db P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:21.950973 10963 tablet_bootstrap.cc:654] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:21.957600 10963 tablet_bootstrap.cc:492] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:21.957787 10963 ts_tablet_manager.cc:1403] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20251025 01:55:21.958681 10963 raft_consensus.cc:359] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.958930 10963 raft_consensus.cc:385] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:21.959023 10963 raft_consensus.cc:740] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:21.939371 11018 raft_consensus.cc:515] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.959342 10963 consensus_queue.cc:260] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:21.999917 10963 ts_tablet_manager.cc:1434] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.042s user 0.001s sys 0.002s
I20251025 01:55:22.000041 11018 leader_election.cc:290] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:22.001199 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d37511aec3746cea09c1ba224cb14db" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:22.001403 10596 raft_consensus.cc:3060] T 3d37511aec3746cea09c1ba224cb14db P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.001952 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d37511aec3746cea09c1ba224cb14db" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:22.002143 10461 raft_consensus.cc:3060] T 3d37511aec3746cea09c1ba224cb14db P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.003631 10596 raft_consensus.cc:2468] T 3d37511aec3746cea09c1ba224cb14db P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:22.004155 10255 leader_election.cc:304] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:22.004473 11018 raft_consensus.cc:2804] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:22.004626 11018 raft_consensus.cc:697] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:22.004904 10461 raft_consensus.cc:2468] T 3d37511aec3746cea09c1ba224cb14db P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:22.004904 11018 consensus_queue.cc:237] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:22.017604 11018 raft_consensus.cc:493] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:22.017774 11018 raft_consensus.cc:515] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:22.018359 11018 leader_election.cc:290] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:22.018707 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "185770378a5b459d9bd420c7fe96c7e1" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:22.018887 10461 raft_consensus.cc:2468] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
W20251025 01:55:22.019135 10255 leader_election.cc:343] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 185770378a5b459d9bd420c7fe96c7e1
I20251025 01:55:22.019349 10255 leader_election.cc:304] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 90109d2fdb464942b20891b5856f8ac2
I20251025 01:55:22.019695 11018 raft_consensus.cc:2804] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:22.019778 11018 raft_consensus.cc:493] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:22.019876 11018 raft_consensus.cc:3060] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.021739 11018 raft_consensus.cc:515] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:22.022244 11018 leader_election.cc:290] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:22.022517 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "185770378a5b459d9bd420c7fe96c7e1" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:22.022696 10461 raft_consensus.cc:3060] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
W20251025 01:55:22.022974 10255 leader_election.cc:343] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 185770378a5b459d9bd420c7fe96c7e1
I20251025 01:55:22.024737 10461 raft_consensus.cc:2468] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:22.025054 10254 leader_election.cc:304] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 90109d2fdb464942b20891b5856f8ac2
I20251025 01:55:22.025297 11018 raft_consensus.cc:2804] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:22.025454 11018 raft_consensus.cc:697] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:22.025669 11018 consensus_queue.cc:237] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
W20251025 01:55:22.069202 10534 connection.cc:537] server connection from 127.0.0.1:43660 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:22.069190 10397 connection.cc:537] server connection from 127.0.0.1:58374 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:22.069216 10256 connection.cc:537] server connection from 127.0.0.1:49356 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:22.069739 10533 connection.cc:537] client connection to 127.9.125.190:35925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:22.069779 10651 connection.cc:537] client connection to 127.9.125.190:35925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:22.070604 10651 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.125.190:35925)
W20251025 01:55:22.070823 10651 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.125.190:35925)
W20251025 01:55:22.070833 10256 connection.cc:537] client connection to 127.9.125.190:35925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:22.072432 10651 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.125.190:35925)
W20251025 01:55:22.072675 10651 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.125.190:35925)
I20251025 01:55:22.167752 11018 raft_consensus.cc:493] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:22.167917 11018 raft_consensus.cc:515] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:22.168542 11018 leader_election.cc:290] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:22.168936 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5906028cad8944bb97e90be8be25003f" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:22.169080 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5906028cad8944bb97e90be8be25003f" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:22.169214 10461 raft_consensus.cc:2468] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:22.169318 10596 raft_consensus.cc:2468] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:22.169687 10254 leader_election.cc:304] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:22.170003 11018 raft_consensus.cc:2804] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:22.170122 11018 raft_consensus.cc:493] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:22.170210 11018 raft_consensus.cc:3060] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.171983 11018 raft_consensus.cc:515] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:22.172447 11018 leader_election.cc:290] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:22.172883 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5906028cad8944bb97e90be8be25003f" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:22.172863 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5906028cad8944bb97e90be8be25003f" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:22.173053 10596 raft_consensus.cc:3060] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.173079 10461 raft_consensus.cc:3060] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.175954 10596 raft_consensus.cc:2468] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:22.175954 10461 raft_consensus.cc:2468] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:22.176412 10255 leader_election.cc:304] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:22.176697 11018 raft_consensus.cc:2804] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:22.176859 11018 raft_consensus.cc:697] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:22.177945 11018 consensus_queue.cc:237] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:22.196262 11001 raft_consensus.cc:493] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:22.196425 11001 raft_consensus.cc:515] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:22.197063 11001 leader_election.cc:290] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:22.197440 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14fdd89953bc49fb91adda5d2a840f3a" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:22.197417 10320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14fdd89953bc49fb91adda5d2a840f3a" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:22.197685 10320 raft_consensus.cc:2393] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in current term 1: Already voted for candidate 90109d2fdb464942b20891b5856f8ac2 in this term.
I20251025 01:55:22.198041 10396 leader_election.cc:304] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c; no voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:55:22.198300 11001 raft_consensus.cc:3060] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.200163 11001 raft_consensus.cc:2749] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:55:22.233681 11047 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 76559300dae44c0d8a17b583a8875b3f)
I20251025 01:55:22.233886 11047 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:22.234012 11048 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 76559300dae44c0d8a17b583a8875b3f)
I20251025 01:55:22.234213 11048 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:22.234565 11047 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:22.236397 11048 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:22.237258 10046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20251025 01:55:22.238452 10046 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
I20251025 01:55:22.238502 10070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "286b4c3025e24a7c8107763fa40d3ab1" candidate_term: 4 candidate_status { last_received { term: 3 index: 79 } } ignore_live_leader: false dest_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" is_pre_election: true
I20251025 01:55:22.238659 10070 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 286b4c3025e24a7c8107763fa40d3ab1 in term 3.
I20251025 01:55:22.239092 10799 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1; no voters:
I20251025 01:55:22.239447 11048 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251025 01:55:22.239477 10799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
I20251025 01:55:22.239573 11048 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 76559300dae44c0d8a17b583a8875b3f)
I20251025 01:55:22.239696 11048 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 3 FOLLOWER]: Advancing to term 4
W20251025 01:55:22.240892 10799 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
I20251025 01:55:22.242358 11048 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:22.243057 11048 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 4 election: Requested vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
I20251025 01:55:22.243753 10070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "286b4c3025e24a7c8107763fa40d3ab1" candidate_term: 4 candidate_status { last_received { term: 3 index: 79 } } ignore_live_leader: false dest_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce"
I20251025 01:55:22.243897 10070 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 3 FOLLOWER]: Advancing to term 4
I20251025 01:55:22.246205 10070 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 286b4c3025e24a7c8107763fa40d3ab1 in term 4.
I20251025 01:55:22.246745 10799 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1; no voters:
I20251025 01:55:22.246999 11048 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 4 FOLLOWER]: Leader election won for term 4
W20251025 01:55:22.247054 10799 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
I20251025 01:55:22.247402 11048 raft_consensus.cc:697] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 4 LEADER]: Becoming Leader. State: Replica: 286b4c3025e24a7c8107763fa40d3ab1, State: Running, Role: LEADER
I20251025 01:55:22.247673 10847 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 79 } } ignore_live_leader: false dest_uuid: "286b4c3025e24a7c8107763fa40d3ab1" is_pre_election: true
I20251025 01:55:22.247897 11048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 3.79, Last appended by leader: 79, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:22.248159 10044 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce; no voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:22.248586 11047 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 01:55:22.251652 11055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286b4c3025e24a7c8107763fa40d3ab1. Latest consensus state: current_term: 4 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:22.251936 11055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:22.252697 11056 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 01:55:22.255623 11056 catalog_manager.cc:679] Loaded metadata for table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a]
I20251025 01:55:22.256075 11056 catalog_manager.cc:679] Loaded metadata for table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b]
I20251025 01:55:22.256454 11056 catalog_manager.cc:679] Loaded metadata for table default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc]
I20251025 01:55:22.256812 11056 catalog_manager.cc:679] Loaded metadata for table default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889]
I20251025 01:55:22.257171 11056 catalog_manager.cc:679] Loaded metadata for table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c]
I20251025 01:55:22.257540 11056 catalog_manager.cc:679] Loaded metadata for table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]
I20251025 01:55:22.257884 11056 catalog_manager.cc:679] Loaded metadata for table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980]
I20251025 01:55:22.258392 11056 catalog_manager.cc:679] Loaded metadata for table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]
I20251025 01:55:22.258749 11056 catalog_manager.cc:679] Loaded metadata for table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f]
I20251025 01:55:22.259088 11056 catalog_manager.cc:679] Loaded metadata for table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]
I20251025 01:55:22.259480 11056 catalog_manager.cc:679] Loaded metadata for table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]
I20251025 01:55:22.260015 11056 catalog_manager.cc:679] Loaded metadata for table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811]
I20251025 01:55:22.260330 11056 catalog_manager.cc:679] Loaded metadata for table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652]
I20251025 01:55:22.260666 11056 catalog_manager.cc:679] Loaded metadata for table default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737]
I20251025 01:55:22.261058 11056 catalog_manager.cc:679] Loaded metadata for table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2]
I20251025 01:55:22.261369 11056 catalog_manager.cc:679] Loaded metadata for table default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0]
I20251025 01:55:22.261651 11056 catalog_manager.cc:679] Loaded metadata for table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a]
I20251025 01:55:22.263788 11056 tablet_loader.cc:96] loaded metadata for tablet 0541393def0b4dcc9cf86038e01d1c21 (table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c])
I20251025 01:55:22.264248 11056 tablet_loader.cc:96] loaded metadata for tablet 14fdd89953bc49fb91adda5d2a840f3a (table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433])
I20251025 01:55:22.264650 11056 tablet_loader.cc:96] loaded metadata for tablet 185770378a5b459d9bd420c7fe96c7e1 (table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980])
I20251025 01:55:22.264912 11056 tablet_loader.cc:96] loaded metadata for tablet 33df938c44154f41b333523a51b65e4e (table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b])
I20251025 01:55:22.265192 11056 tablet_loader.cc:96] loaded metadata for tablet 3d37511aec3746cea09c1ba224cb14db (table default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0])
I20251025 01:55:22.265650 11056 tablet_loader.cc:96] loaded metadata for tablet 5906028cad8944bb97e90be8be25003f (table default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889])
I20251025 01:55:22.265924 11056 tablet_loader.cc:96] loaded metadata for tablet 5a2f9794447f4d7e8f6f548e4b26c8c6 (table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b])
I20251025 01:55:22.266408 11056 tablet_loader.cc:96] loaded metadata for tablet 898ce3e93e56489f83a04fc46e7328de (table default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc])
I20251025 01:55:22.267727 11056 tablet_loader.cc:96] loaded metadata for tablet a0722d87a2814e589a20a0e4cb274c97 (table default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737])
I20251025 01:55:22.268041 11056 tablet_loader.cc:96] loaded metadata for tablet a1269807c3644c629ed5d0e4e45c5efc (table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a])
I20251025 01:55:22.268378 11056 tablet_loader.cc:96] loaded metadata for tablet a8bb9f9752b641499955e678b2ccc943 (table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811])
I20251025 01:55:22.268676 11056 tablet_loader.cc:96] loaded metadata for tablet b3578b8a1d8f412d9a8b2b968bb9c583 (table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2])
I20251025 01:55:22.269357 11056 tablet_loader.cc:96] loaded metadata for tablet cceaea14d4054de09cdaacb93f9b5582 (table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f])
I20251025 01:55:22.269667 11056 tablet_loader.cc:96] loaded metadata for tablet ce356b2a74154595bafdc91c27767b09 (table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a])
I20251025 01:55:22.269949 11056 tablet_loader.cc:96] loaded metadata for tablet d7df9d477cf1448d91cd8cac0ab321a9 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5])
I20251025 01:55:22.270421 11056 tablet_loader.cc:96] loaded metadata for tablet dcf81923a06d4750bbec550cff9bd91d (table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812])
I20251025 01:55:22.270917 11056 tablet_loader.cc:96] loaded metadata for tablet f13a1ad12e2f4f2c95103cf8d76d6da6 (table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652])
I20251025 01:55:22.271162 11056 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 01:55:22.271970 11018 raft_consensus.cc:493] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:22.272130 11018 raft_consensus.cc:515] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:22.272605 11056 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:22.272698 11056 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 01:55:22.272717 11018 leader_election.cc:290] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:22.273146 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "898ce3e93e56489f83a04fc46e7328de" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:22.273363 10461 raft_consensus.cc:2468] T 898ce3e93e56489f83a04fc46e7328de P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:22.273388 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "898ce3e93e56489f83a04fc46e7328de" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:22.273607 10596 raft_consensus.cc:2468] T 898ce3e93e56489f83a04fc46e7328de P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:22.273839 10254 leader_election.cc:304] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:22.274119 11018 raft_consensus.cc:2804] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:22.274217 11018 raft_consensus.cc:493] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:22.274313 11018 raft_consensus.cc:3060] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.274770 11056 catalog_manager.cc:1514] Loading token signing keys...
I20251025 01:55:22.275861 11056 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Loaded TSK: 0
I20251025 01:55:22.276139 11018 raft_consensus.cc:515] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:22.276614 11018 leader_election.cc:290] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:22.276846 11056 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 01:55:22.277040 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "898ce3e93e56489f83a04fc46e7328de" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:22.277144 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "898ce3e93e56489f83a04fc46e7328de" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:22.277244 10461 raft_consensus.cc:3060] T 898ce3e93e56489f83a04fc46e7328de P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.277355 10596 raft_consensus.cc:3060] T 898ce3e93e56489f83a04fc46e7328de P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.278750 11056 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251025 01:55:22.279168 10596 raft_consensus.cc:2468] T 898ce3e93e56489f83a04fc46e7328de P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:22.279577 10255 leader_election.cc:304] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:22.279807 11018 raft_consensus.cc:2804] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:22.279978 11018 raft_consensus.cc:697] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:22.280167 10461 raft_consensus.cc:2468] T 898ce3e93e56489f83a04fc46e7328de P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:22.280242 11018 consensus_queue.cc:237] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:22.281217 11056 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: 23
I20251025 01:55:22.284274 10825 catalog_manager.cc:5649] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b reported cstate change: term changed from 0 to 1, leader changed from <none> to c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129). New cstate: current_term: 1 leader_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20251025 01:55:22.290941 10070 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Refusing update from remote peer 286b4c3025e24a7c8107763fa40d3ab1: Log matching property violated. Preceding OpId in replica: term: 3 index: 79. Preceding OpId from leader: term: 4 index: 81. (index mismatch)
W20251025 01:55:22.291702 10799 consensus_peers.cc:597] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 -> Peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Couldn't send request to peer 76559300dae44c0d8a17b583a8875b3f. Status: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 01:55:22.291757 11048 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20251025 01:55:22.295023 11047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286b4c3025e24a7c8107763fa40d3ab1. Latest consensus state: current_term: 4 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:22.295243 11047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:22.296578 10831 master_service.cc:954] ReplaceTablet: received request to replace tablet 185770378a5b459d9bd420c7fe96c7e1 from {username='slave'} at 127.0.0.1:34110
I20251025 01:55:22.297571 10830 master_service.cc:954] ReplaceTablet: received request to replace tablet a1269807c3644c629ed5d0e4e45c5efc from {username='slave'} at 127.0.0.1:34110
I20251025 01:55:22.297806 10828 master_service.cc:954] ReplaceTablet: received request to replace tablet 14fdd89953bc49fb91adda5d2a840f3a from {username='slave'} at 127.0.0.1:34110
I20251025 01:55:22.298859 10821 master_service.cc:954] ReplaceTablet: received request to replace tablet b8f17943e2d0462da38dcdfb4b1cc025 from {username='slave'} at 127.0.0.1:34110
I20251025 01:55:22.301525 11047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:22.301723 11047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:22.308344 10829 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34110:
name: "default.Table_9391ddd403f54e168c61fa879d7f8f6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:22.308581 10817 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34110:
name: "default.Table_356101dc020f46c490a4bbc739447d43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:22.309283 10821 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34110:
name: "default.Table_50c0606a959945b3a1edd9df467a65dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:22.309580 10824 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34110:
name: "default.Table_d93bab5cc39a448caf73455685fb742f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:22.310119 10825 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34110:
name: "default.Table_9328a3e2d017427796e74bbabc9f07bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:22.311594 11048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:22.311100 10822 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34110:
name: "default.Table_b54f3b1b46af4568a75e8141684d8f97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:22.311906 11048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:22.311411 10812 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34110:
name: "default.Table_1a02719f7e0145b28ac84c3a43091e4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:22.316238 10828 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Sending DeleteTablet for 3 replicas of tablet 14fdd89953bc49fb91adda5d2a840f3a
I20251025 01:55:22.318279 10828 catalog_manager.cc:6719] ReplaceTablet: tablet 14fdd89953bc49fb91adda5d2a840f3a deleted and replaced by tablet effc501d8ef24438a3fd7c30feb32f87
F20251025 01:55:22.323915 10906 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 4 out of 13 received events: NotificationEvent(eventId=24, eventTime=1761357321, eventType=CREATE_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_a981c91dba574e1c8ccce55a41c4bd88, message={"server":"","servicePrincipal":"","db":"default","table":"table_a981c91dba574e1c8ccce55a41c4bd88","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a981c91dba574e1c8ccce55a41c4bd88\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357320},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_a981c91dba574e1c8ccce55a41c4bd88\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a981c91dba574e1c8ccce55a41c4bd88\",\"kudu.table_id\":\"269cfd64fbd94f0486843525917d9bb4\",\"transient_lastDdlTime\":\"1761357320\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_44e4ccc4f2de47198a0599e2577108cf, message={"server":"","servicePrincipal":"","db":"default","table":"table_44e4ccc4f2de47198a0599e2577108cf","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_44e4ccc4f2de47198a0599e2577108cf\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357319},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_44e4ccc4f2de47198a0599e2577108cf\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_44e4ccc4f2de47198a0599e2577108cf\",\"kudu.table_id\":\"58afab4104e84c868eeccda411b8452b\",\"transient_lastDdlTime\":\"1761357319\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251025 01:55:22.325611 10828 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34110:
name: "default.Table_ac8050cb49d54bc2a444d81cf8492c75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f891c26adcd google::LogMessage::Fail() at ??:0
@ 0x7f891c26eb93 google::LogMessage::SendToLog() at ??:0
I20251025 01:55:22.333070 10830 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Sending DeleteTablet for 3 replicas of tablet a1269807c3644c629ed5d0e4e45c5efc
@ 0x7f891c26a7cc google::LogMessage::Flush() at ??:0
I20251025 01:55:22.333702 10830 catalog_manager.cc:6719] ReplaceTablet: tablet a1269807c3644c629ed5d0e4e45c5efc deleted and replaced by tablet 87aaf87a6c284ca980e5fb6f896c5e4e
I20251025 01:55:22.334905 10831 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Sending DeleteTablet for 3 replicas of tablet 185770378a5b459d9bd420c7fe96c7e1
@ 0x7f891c26aa11 google::LogMessage::~LogMessage() at ??:0
I20251025 01:55:22.335511 10831 catalog_manager.cc:6719] ReplaceTablet: tablet 185770378a5b459d9bd420c7fe96c7e1 deleted and replaced by tablet 174f2bfde6ef4ab4b90750c0f63fa9a1
I20251025 01:55:22.336581 10441 tablet_service.cc:1552] Processing DeleteTablet for tablet 14fdd89953bc49fb91adda5d2a840f3a with delete_type TABLET_DATA_DELETED (replaced by tablet effc501d8ef24438a3fd7c30feb32f87) from {username='slave'} at 127.0.0.1:58386
I20251025 01:55:22.336864 10440 tablet_service.cc:1552] Processing DeleteTablet for tablet a1269807c3644c629ed5d0e4e45c5efc with delete_type TABLET_DATA_DELETED (replaced by tablet 87aaf87a6c284ca980e5fb6f896c5e4e) from {username='slave'} at 127.0.0.1:58386
I20251025 01:55:22.337075 10439 tablet_service.cc:1552] Processing DeleteTablet for tablet 185770378a5b459d9bd420c7fe96c7e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 174f2bfde6ef4ab4b90750c0f63fa9a1) from {username='slave'} at 127.0.0.1:58386
I20251025 01:55:22.337282 10949 tablet_replica.cc:333] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:22.337507 10949 raft_consensus.cc:2243] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:22.337678 10949 raft_consensus.cc:2272] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:22.338483 10949 ts_tablet_manager.cc:1916] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:22.344211 10578 tablet_service.cc:1552] Processing DeleteTablet for tablet 14fdd89953bc49fb91adda5d2a840f3a with delete_type TABLET_DATA_DELETED (replaced by tablet effc501d8ef24438a3fd7c30feb32f87) from {username='slave'} at 127.0.0.1:43676
I20251025 01:55:22.344496 10947 tablet_replica.cc:333] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:22.344722 10947 raft_consensus.cc:2243] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:22.344991 10947 pending_rounds.cc:70] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2: Trying to abort 1 pending ops.
I20251025 01:55:22.345067 10947 pending_rounds.cc:77] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214519590705254400 op_type: NO_OP noop_request { }
I20251025 01:55:22.345158 10299 tablet_service.cc:1552] Processing DeleteTablet for tablet 185770378a5b459d9bd420c7fe96c7e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 174f2bfde6ef4ab4b90750c0f63fa9a1) from {username='slave'} at 127.0.0.1:49362
I20251025 01:55:22.345187 10947 raft_consensus.cc:2889] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 01:55:22.345289 10947 raft_consensus.cc:2272] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:22.345558 10301 tablet_service.cc:1552] Processing DeleteTablet for tablet 14fdd89953bc49fb91adda5d2a840f3a with delete_type TABLET_DATA_DELETED (replaced by tablet effc501d8ef24438a3fd7c30feb32f87) from {username='slave'} at 127.0.0.1:49362
I20251025 01:55:22.345765 10300 tablet_service.cc:1552] Processing DeleteTablet for tablet a1269807c3644c629ed5d0e4e45c5efc with delete_type TABLET_DATA_DELETED (replaced by tablet 87aaf87a6c284ca980e5fb6f896c5e4e) from {username='slave'} at 127.0.0.1:49362
I20251025 01:55:22.345969 10946 tablet_replica.cc:333] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:22.346165 10946 raft_consensus.cc:2243] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:22.346408 10946 pending_rounds.cc:70] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b: Trying to abort 1 pending ops.
I20251025 01:55:22.346482 10946 pending_rounds.cc:77] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214519591018635264 op_type: NO_OP noop_request { }
I20251025 01:55:22.346601 10946 raft_consensus.cc:2889] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 01:55:22.346724 10946 raft_consensus.cc:2272] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:22.344466 10576 tablet_service.cc:1552] Processing DeleteTablet for tablet a1269807c3644c629ed5d0e4e45c5efc with delete_type TABLET_DATA_DELETED (replaced by tablet 87aaf87a6c284ca980e5fb6f896c5e4e) from {username='slave'} at 127.0.0.1:43676
I20251025 01:55:22.347558 10578 tablet_service.cc:1552] Processing DeleteTablet for tablet 185770378a5b459d9bd420c7fe96c7e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 174f2bfde6ef4ab4b90750c0f63fa9a1) from {username='slave'} at 127.0.0.1:43676
I20251025 01:55:22.348158 10946 ts_tablet_manager.cc:1916] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:22.351737 10947 ts_tablet_manager.cc:1916] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:22.351930 10949 ts_tablet_manager.cc:1929] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:22.352065 10949 log.cc:1199] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/14fdd89953bc49fb91adda5d2a840f3a
I20251025 01:55:22.352418 10949 ts_tablet_manager.cc:1950] T 14fdd89953bc49fb91adda5d2a840f3a P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:22.353312 10949 tablet_replica.cc:333] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:22.353477 10949 raft_consensus.cc:2243] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:22.353618 10797 catalog_manager.cc:4985] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): tablet 14fdd89953bc49fb91adda5d2a840f3a (table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]) successfully deleted
I20251025 01:55:22.353816 10949 raft_consensus.cc:2272] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:22.354282 10949 ts_tablet_manager.cc:1916] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:22.360754 10831 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34110:
name: "default.Table_20c488ac6bf24db48ea2ab061d253e7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:22.361281 10830 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34110:
name: "default.Table_a981c91dba574e1c8ccce55a41c4bd88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f8932c23b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251025 01:55:22.362414 10947 ts_tablet_manager.cc:1929] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:22.362516 10947 log.cc:1199] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/14fdd89953bc49fb91adda5d2a840f3a
I20251025 01:55:22.362864 10947 ts_tablet_manager.cc:1950] T 14fdd89953bc49fb91adda5d2a840f3a P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:22.363844 10798 catalog_manager.cc:4985] TS 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): tablet 14fdd89953bc49fb91adda5d2a840f3a (table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]) successfully deleted
I20251025 01:55:22.364635 10946 ts_tablet_manager.cc:1929] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:22.364746 10946 log.cc:1199] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/185770378a5b459d9bd420c7fe96c7e1
I20251025 01:55:22.365139 10946 ts_tablet_manager.cc:1950] T 185770378a5b459d9bd420c7fe96c7e1 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:22.366180 10798 catalog_manager.cc:4985] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): tablet 185770378a5b459d9bd420c7fe96c7e1 (table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980]) successfully deleted
I20251025 01:55:22.366477 10947 tablet_replica.cc:333] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:22.366668 10947 raft_consensus.cc:2243] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:22.366802 10947 raft_consensus.cc:2272] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:22.367417 10946 tablet_replica.cc:333] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:22.367609 10946 raft_consensus.cc:2243] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:22.367635 10949 ts_tablet_manager.cc:1929] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:22.367728 10949 log.cc:1199] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/a1269807c3644c629ed5d0e4e45c5efc
I20251025 01:55:22.367759 10946 raft_consensus.cc:2272] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:22.368090 10949 ts_tablet_manager.cc:1950] T a1269807c3644c629ed5d0e4e45c5efc P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:22.368305 10946 ts_tablet_manager.cc:1916] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:22.368889 10949 tablet_replica.cc:333] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:22.369071 10949 raft_consensus.cc:2243] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:22.369061 10797 catalog_manager.cc:4985] TS 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): tablet a1269807c3644c629ed5d0e4e45c5efc (table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a]) successfully deleted
I20251025 01:55:22.369206 10949 raft_consensus.cc:2272] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f8932c22411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251025 01:55:22.369915 10949 ts_tablet_manager.cc:1916] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:22.371639 10947 ts_tablet_manager.cc:1916] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f8932c2a08c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251025 01:55:22.375327 10578 tablet_service.cc:1505] Processing CreateTablet for tablet 33df938c44154f41b333523a51b65e4e (DEFAULT_TABLE table=default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:22.375720 10578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33df938c44154f41b333523a51b65e4e. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f8932c29f09 std::_Function_handler<>::_M_invoke() at ??:0
I20251025 01:55:22.377951 10301 tablet_service.cc:1505] Processing CreateTablet for tablet effc501d8ef24438a3fd7c30feb32f87 (DEFAULT_TABLE table=default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0xa8867d std::function<>::operator()() at ??:?
I20251025 01:55:22.378418 10301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet effc501d8ef24438a3fd7c30feb32f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:22.378408 10576 tablet_service.cc:1505] Processing CreateTablet for tablet effc501d8ef24438a3fd7c30feb32f87 (DEFAULT_TABLE table=default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:22.378758 10576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet effc501d8ef24438a3fd7c30feb32f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:22.381312 10297 tablet_service.cc:1505] Processing CreateTablet for tablet a0722d87a2814e589a20a0e4cb274c97 (DEFAULT_TABLE table=default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:22.381703 10297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0722d87a2814e589a20a0e4cb274c97. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f891efdeee9 kudu::Thread::SuperviseThread() at ??:0
I20251025 01:55:22.383525 10300 tablet_service.cc:1505] Processing CreateTablet for tablet 33df938c44154f41b333523a51b65e4e (DEFAULT_TABLE table=default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:22.383838 10300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33df938c44154f41b333523a51b65e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:22.385186 10575 tablet_service.cc:1505] Processing CreateTablet for tablet 5a2f9794447f4d7e8f6f548e4b26c8c6 (DEFAULT_TABLE table=default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f89225e66db start_thread at ??:0
I20251025 01:55:22.385597 10575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a2f9794447f4d7e8f6f548e4b26c8c6. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f891b11b71f clone at ??:0
I20251025 01:55:22.387518 10298 tablet_service.cc:1505] Processing CreateTablet for tablet 5a2f9794447f4d7e8f6f548e4b26c8c6 (DEFAULT_TABLE table=default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:22.377974 10299 tablet_service.cc:1505] Processing CreateTablet for tablet d7df9d477cf1448d91cd8cac0ab321a9 (DEFAULT_TABLE table=default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:22.387773 10439 tablet_service.cc:1505] Processing CreateTablet for tablet 33df938c44154f41b333523a51b65e4e (DEFAULT_TABLE table=default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:22.387852 10298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a2f9794447f4d7e8f6f548e4b26c8c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:22.387991 10299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7df9d477cf1448d91cd8cac0ab321a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:22.388134 10439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33df938c44154f41b333523a51b65e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:22.388666 10949 ts_tablet_manager.cc:1929] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:22.388804 10949 log.cc:1199] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/185770378a5b459d9bd420c7fe96c7e1
I20251025 01:55:22.389133 10949 ts_tablet_manager.cc:1950] T 185770378a5b459d9bd420c7fe96c7e1 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:22.390578 10574 tablet_service.cc:1505] Processing CreateTablet for tablet a0722d87a2814e589a20a0e4cb274c97 (DEFAULT_TABLE table=default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:22.391019 10574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0722d87a2814e589a20a0e4cb274c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:22.392042 10442 tablet_service.cc:1505] Processing CreateTablet for tablet 5a2f9794447f4d7e8f6f548e4b26c8c6 (DEFAULT_TABLE table=default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:22.392338 10442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a2f9794447f4d7e8f6f548e4b26c8c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:22.394179 10959 tablet_bootstrap.cc:492] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:22.394490 10577 tablet_service.cc:1505] Processing CreateTablet for tablet d7df9d477cf1448d91cd8cac0ab321a9 (DEFAULT_TABLE table=default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:22.394806 10577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7df9d477cf1448d91cd8cac0ab321a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:22.395900 10959 tablet_bootstrap.cc:654] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.398180 10441 tablet_service.cc:1505] Processing CreateTablet for tablet d7df9d477cf1448d91cd8cac0ab321a9 (DEFAULT_TABLE table=default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:22.398275 10437 tablet_service.cc:1505] Processing CreateTablet for tablet a0722d87a2814e589a20a0e4cb274c97 (DEFAULT_TABLE table=default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:22.398527 10441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7df9d477cf1448d91cd8cac0ab321a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:22.398759 10437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0722d87a2814e589a20a0e4cb274c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:22.399015 10440 tablet_service.cc:1505] Processing CreateTablet for tablet effc501d8ef24438a3fd7c30feb32f87 (DEFAULT_TABLE table=default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:22.399370 10440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet effc501d8ef24438a3fd7c30feb32f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:22.407469 10963 tablet_bootstrap.cc:492] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:22.409013 10963 tablet_bootstrap.cc:654] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.410054 10959 tablet_bootstrap.cc:492] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:22.410174 10959 ts_tablet_manager.cc:1403] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20251025 01:55:22.410935 10959 raft_consensus.cc:359] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.411096 10959 raft_consensus.cc:385] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.411175 10959 raft_consensus.cc:740] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.411427 10959 consensus_queue.cc:260] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.411815 10369 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
I20251025 01:55:22.414458 10959 ts_tablet_manager.cc:1434] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 01:55:22.414839 10959 tablet_bootstrap.cc:492] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:22.416368 10959 tablet_bootstrap.cc:654] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.420379 10959 tablet_bootstrap.cc:492] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:22.420384 11074 tablet_bootstrap.cc:492] T a0722d87a2814e589a20a0e4cb274c97 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:22.420547 10959 ts_tablet_manager.cc:1403] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251025 01:55:22.420866 10963 tablet_bootstrap.cc:492] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:22.421000 10963 ts_tablet_manager.cc:1403] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20251025 01:55:22.421228 10959 raft_consensus.cc:359] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.421396 10959 raft_consensus.cc:385] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.421473 10959 raft_consensus.cc:740] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.421679 10959 consensus_queue.cc:260] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.421838 10963 raft_consensus.cc:359] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.422025 10963 raft_consensus.cc:385] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.422060 11074 tablet_bootstrap.cc:654] T a0722d87a2814e589a20a0e4cb274c97 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.423167 10959 ts_tablet_manager.cc:1434] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 01:55:22.423703 10959 tablet_bootstrap.cc:492] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:22.424398 10963 raft_consensus.cc:740] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.424611 10963 consensus_queue.cc:260] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.427469 10963 ts_tablet_manager.cc:1434] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20251025 01:55:22.427870 10963 tablet_bootstrap.cc:492] T d7df9d477cf1448d91cd8cac0ab321a9 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:22.429684 10963 tablet_bootstrap.cc:654] T d7df9d477cf1448d91cd8cac0ab321a9 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.430734 10959 tablet_bootstrap.cc:654] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.437645 10963 tablet_bootstrap.cc:492] T d7df9d477cf1448d91cd8cac0ab321a9 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:22.437760 10963 ts_tablet_manager.cc:1403] T d7df9d477cf1448d91cd8cac0ab321a9 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20251025 01:55:22.438333 10963 raft_consensus.cc:359] T d7df9d477cf1448d91cd8cac0ab321a9 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.438509 10963 raft_consensus.cc:385] T d7df9d477cf1448d91cd8cac0ab321a9 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.438560 10963 raft_consensus.cc:740] T d7df9d477cf1448d91cd8cac0ab321a9 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.438769 10963 consensus_queue.cc:260] T d7df9d477cf1448d91cd8cac0ab321a9 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.440054 10963 ts_tablet_manager.cc:1434] T d7df9d477cf1448d91cd8cac0ab321a9 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 01:55:22.440393 10963 tablet_bootstrap.cc:492] T a0722d87a2814e589a20a0e4cb274c97 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:22.442540 10963 tablet_bootstrap.cc:654] T a0722d87a2814e589a20a0e4cb274c97 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.447381 11074 tablet_bootstrap.cc:492] T a0722d87a2814e589a20a0e4cb274c97 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:22.447511 11074 ts_tablet_manager.cc:1403] T a0722d87a2814e589a20a0e4cb274c97 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.017s
I20251025 01:55:22.448179 10947 ts_tablet_manager.cc:1929] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:22.448307 10947 log.cc:1199] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/a1269807c3644c629ed5d0e4e45c5efc
I20251025 01:55:22.448230 11074 raft_consensus.cc:359] T a0722d87a2814e589a20a0e4cb274c97 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.448386 11074 raft_consensus.cc:385] T a0722d87a2814e589a20a0e4cb274c97 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.448482 11074 raft_consensus.cc:740] T a0722d87a2814e589a20a0e4cb274c97 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.448654 10947 ts_tablet_manager.cc:1950] T a1269807c3644c629ed5d0e4e45c5efc P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:22.448725 11074 consensus_queue.cc:260] T a0722d87a2814e589a20a0e4cb274c97 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.451644 10946 ts_tablet_manager.cc:1929] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 01:55:22.451773 10946 log.cc:1199] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/14fdd89953bc49fb91adda5d2a840f3a
I20251025 01:55:22.452101 10946 ts_tablet_manager.cc:1950] T 14fdd89953bc49fb91adda5d2a840f3a P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:22.452600 11074 ts_tablet_manager.cc:1434] T a0722d87a2814e589a20a0e4cb274c97 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 01:55:22.452975 10946 tablet_replica.cc:333] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:22.453154 10946 raft_consensus.cc:2243] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:22.453394 10946 pending_rounds.cc:70] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b: Trying to abort 1 pending ops.
I20251025 01:55:22.453455 10946 pending_rounds.cc:77] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214519589878587392 op_type: NO_OP noop_request { }
I20251025 01:55:22.453581 10946 raft_consensus.cc:2889] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 01:55:22.453691 10946 raft_consensus.cc:2272] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:22.454974 10959 tablet_bootstrap.cc:492] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:22.455098 10959 ts_tablet_manager.cc:1403] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.004s
I20251025 01:55:22.455888 11074 tablet_bootstrap.cc:492] T 33df938c44154f41b333523a51b65e4e P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:22.456125 10959 raft_consensus.cc:359] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.456270 10959 raft_consensus.cc:385] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.456350 10959 raft_consensus.cc:740] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.456553 10959 consensus_queue.cc:260] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.457095 10959 ts_tablet_manager.cc:1434] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 01:55:22.457314 10959 tablet_bootstrap.cc:492] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:22.458571 11074 tablet_bootstrap.cc:654] T 33df938c44154f41b333523a51b65e4e P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.458778 10959 tablet_bootstrap.cc:654] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.460479 10946 ts_tablet_manager.cc:1916] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:22.465585 10963 tablet_bootstrap.cc:492] T a0722d87a2814e589a20a0e4cb274c97 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:22.465714 10963 ts_tablet_manager.cc:1403] T a0722d87a2814e589a20a0e4cb274c97 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.001s
I20251025 01:55:22.466413 10963 raft_consensus.cc:359] T a0722d87a2814e589a20a0e4cb274c97 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.466559 10963 raft_consensus.cc:385] T a0722d87a2814e589a20a0e4cb274c97 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.466624 10963 raft_consensus.cc:740] T a0722d87a2814e589a20a0e4cb274c97 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.466836 10963 consensus_queue.cc:260] T a0722d87a2814e589a20a0e4cb274c97 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.467008 10946 ts_tablet_manager.cc:1929] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:22.467121 10946 log.cc:1199] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/a1269807c3644c629ed5d0e4e45c5efc
I20251025 01:55:22.467494 10946 ts_tablet_manager.cc:1950] T a1269807c3644c629ed5d0e4e45c5efc P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:22.468452 10963 ts_tablet_manager.cc:1434] T a0722d87a2814e589a20a0e4cb274c97 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 01:55:22.468833 10963 tablet_bootstrap.cc:492] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:22.471710 10963 tablet_bootstrap.cc:654] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.475059 11074 tablet_bootstrap.cc:492] T 33df938c44154f41b333523a51b65e4e P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:22.475229 11074 ts_tablet_manager.cc:1403] T 33df938c44154f41b333523a51b65e4e P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20251025 01:55:22.475904 11074 raft_consensus.cc:359] T 33df938c44154f41b333523a51b65e4e P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:22.476075 11074 raft_consensus.cc:385] T 33df938c44154f41b333523a51b65e4e P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.476148 11074 raft_consensus.cc:740] T 33df938c44154f41b333523a51b65e4e P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.476310 10959 tablet_bootstrap.cc:492] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:22.476462 10959 ts_tablet_manager.cc:1403] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.009s
I20251025 01:55:22.476408 11074 consensus_queue.cc:260] T 33df938c44154f41b333523a51b65e4e P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:22.477188 10959 raft_consensus.cc:359] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.477356 10959 raft_consensus.cc:385] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.477418 10959 raft_consensus.cc:740] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.477619 10959 consensus_queue.cc:260] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.478096 10959 ts_tablet_manager.cc:1434] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 01:55:22.478395 10959 tablet_bootstrap.cc:492] T 33df938c44154f41b333523a51b65e4e P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:22.479925 10959 tablet_bootstrap.cc:654] T 33df938c44154f41b333523a51b65e4e P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.484068 11074 ts_tablet_manager.cc:1434] T 33df938c44154f41b333523a51b65e4e P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251025 01:55:22.484405 10963 tablet_bootstrap.cc:492] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:22.484546 10963 ts_tablet_manager.cc:1403] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.004s
I20251025 01:55:22.485284 10963 raft_consensus.cc:359] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:22.485468 10963 raft_consensus.cc:385] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.485531 10963 raft_consensus.cc:740] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.485783 10963 consensus_queue.cc:260] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:22.487295 10963 ts_tablet_manager.cc:1434] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 01:55:22.487713 10963 tablet_bootstrap.cc:492] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:22.489219 10963 tablet_bootstrap.cc:654] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.493145 11074 tablet_bootstrap.cc:492] T effc501d8ef24438a3fd7c30feb32f87 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:22.495256 11074 tablet_bootstrap.cc:654] T effc501d8ef24438a3fd7c30feb32f87 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.497335 10959 tablet_bootstrap.cc:492] T 33df938c44154f41b333523a51b65e4e P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:22.497352 11074 tablet_bootstrap.cc:492] T effc501d8ef24438a3fd7c30feb32f87 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:22.497445 11074 ts_tablet_manager.cc:1403] T effc501d8ef24438a3fd7c30feb32f87 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251025 01:55:22.497468 10959 ts_tablet_manager.cc:1403] T 33df938c44154f41b333523a51b65e4e P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20251025 01:55:22.497927 11074 raft_consensus.cc:359] T effc501d8ef24438a3fd7c30feb32f87 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.498113 11074 raft_consensus.cc:385] T effc501d8ef24438a3fd7c30feb32f87 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.498171 11074 raft_consensus.cc:740] T effc501d8ef24438a3fd7c30feb32f87 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.498103 10959 raft_consensus.cc:359] T 33df938c44154f41b333523a51b65e4e P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:22.498248 10959 raft_consensus.cc:385] T 33df938c44154f41b333523a51b65e4e P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.498320 11074 consensus_queue.cc:260] T effc501d8ef24438a3fd7c30feb32f87 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.498346 10959 raft_consensus.cc:740] T 33df938c44154f41b333523a51b65e4e P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.498582 10959 consensus_queue.cc:260] T 33df938c44154f41b333523a51b65e4e P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:22.499032 10959 ts_tablet_manager.cc:1434] T 33df938c44154f41b333523a51b65e4e P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 01:55:22.500520 11074 ts_tablet_manager.cc:1434] T effc501d8ef24438a3fd7c30feb32f87 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 01:55:22.500900 11074 tablet_bootstrap.cc:492] T d7df9d477cf1448d91cd8cac0ab321a9 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:22.502354 10963 tablet_bootstrap.cc:492] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:22.502487 10963 ts_tablet_manager.cc:1403] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20251025 01:55:22.503190 10963 raft_consensus.cc:359] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.503366 10963 raft_consensus.cc:385] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.503454 10963 raft_consensus.cc:740] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.503652 10963 consensus_queue.cc:260] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.503749 11074 tablet_bootstrap.cc:654] T d7df9d477cf1448d91cd8cac0ab321a9 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.504063 10963 ts_tablet_manager.cc:1434] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 01:55:22.511677 11074 tablet_bootstrap.cc:492] T d7df9d477cf1448d91cd8cac0ab321a9 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:22.511811 11074 ts_tablet_manager.cc:1403] T d7df9d477cf1448d91cd8cac0ab321a9 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20251025 01:55:22.512552 11074 raft_consensus.cc:359] T d7df9d477cf1448d91cd8cac0ab321a9 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.512744 11074 raft_consensus.cc:385] T d7df9d477cf1448d91cd8cac0ab321a9 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.512802 11074 raft_consensus.cc:740] T d7df9d477cf1448d91cd8cac0ab321a9 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.513001 11074 consensus_queue.cc:260] T d7df9d477cf1448d91cd8cac0ab321a9 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.514214 11074 ts_tablet_manager.cc:1434] T d7df9d477cf1448d91cd8cac0ab321a9 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 01:55:22.514966 11074 tablet_bootstrap.cc:492] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:22.518070 11074 tablet_bootstrap.cc:654] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:22.519944 11074 tablet_bootstrap.cc:492] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:22.520057 11074 ts_tablet_manager.cc:1403] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20251025 01:55:22.520718 11074 raft_consensus.cc:359] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.520875 11074 raft_consensus.cc:385] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:22.520954 11074 raft_consensus.cc:740] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:22.521184 11074 consensus_queue.cc:260] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.521829 11074 ts_tablet_manager.cc:1434] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 01:55:22.550925 11018 raft_consensus.cc:493] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:22.551081 11018 raft_consensus.cc:515] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.551769 11018 leader_election.cc:290] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:22.552243 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a2f9794447f4d7e8f6f548e4b26c8c6" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:22.552287 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a2f9794447f4d7e8f6f548e4b26c8c6" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:22.552496 10596 raft_consensus.cc:2468] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:22.552516 10461 raft_consensus.cc:2468] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:22.552963 10254 leader_election.cc:304] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:22.553293 11018 raft_consensus.cc:2804] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:22.553435 11018 raft_consensus.cc:493] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:22.553542 11018 raft_consensus.cc:3060] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.556593 11018 raft_consensus.cc:515] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.557219 11018 leader_election.cc:290] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:22.557492 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a2f9794447f4d7e8f6f548e4b26c8c6" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:22.557694 10596 raft_consensus.cc:3060] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.557821 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a2f9794447f4d7e8f6f548e4b26c8c6" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:22.558038 10461 raft_consensus.cc:3060] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.560614 10596 raft_consensus.cc:2468] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:22.560907 10461 raft_consensus.cc:2468] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:22.561026 10255 leader_election.cc:304] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:22.561321 11018 raft_consensus.cc:2804] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:22.561486 11018 raft_consensus.cc:697] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:22.561815 11018 consensus_queue.cc:237] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.608033 11001 raft_consensus.cc:493] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:22.608243 11001 raft_consensus.cc:515] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.608913 11001 leader_election.cc:290] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:22.609544 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "effc501d8ef24438a3fd7c30feb32f87" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:22.609539 10320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "effc501d8ef24438a3fd7c30feb32f87" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:22.609783 10320 raft_consensus.cc:2468] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 0.
I20251025 01:55:22.609838 10596 raft_consensus.cc:2468] T effc501d8ef24438a3fd7c30feb32f87 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 0.
I20251025 01:55:22.610265 10396 leader_election.cc:304] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2; no voters:
I20251025 01:55:22.610564 11001 raft_consensus.cc:2804] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:22.610651 11001 raft_consensus.cc:493] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:22.610750 11001 raft_consensus.cc:3060] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.612684 11001 raft_consensus.cc:515] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.613169 11001 leader_election.cc:290] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 election: Requested vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:22.613622 10320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "effc501d8ef24438a3fd7c30feb32f87" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
I20251025 01:55:22.613623 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "effc501d8ef24438a3fd7c30feb32f87" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:22.613835 10320 raft_consensus.cc:3060] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.613835 10596 raft_consensus.cc:3060] T effc501d8ef24438a3fd7c30feb32f87 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.616822 10320 raft_consensus.cc:2468] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 1.
I20251025 01:55:22.616827 10596 raft_consensus.cc:2468] T effc501d8ef24438a3fd7c30feb32f87 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 1.
I20251025 01:55:22.617228 10396 leader_election.cc:304] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:22.617496 11001 raft_consensus.cc:2804] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:22.618009 11001 raft_consensus.cc:697] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 LEADER]: Becoming Leader. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Running, Role: LEADER
I20251025 01:55:22.618508 11001 consensus_queue.cc:237] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.635444 11018 raft_consensus.cc:493] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:22.635635 11018 raft_consensus.cc:515] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.636341 11018 leader_election.cc:290] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:22.636842 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7df9d477cf1448d91cd8cac0ab321a9" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:22.637059 10596 raft_consensus.cc:2468] T d7df9d477cf1448d91cd8cac0ab321a9 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:22.637034 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7df9d477cf1448d91cd8cac0ab321a9" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:22.637226 10461 raft_consensus.cc:2468] T d7df9d477cf1448d91cd8cac0ab321a9 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:22.637508 10255 leader_election.cc:304] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:22.637804 11018 raft_consensus.cc:2804] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:22.637900 11018 raft_consensus.cc:493] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:22.637989 11018 raft_consensus.cc:3060] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.640944 11018 raft_consensus.cc:515] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.641940 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7df9d477cf1448d91cd8cac0ab321a9" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:22.641999 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7df9d477cf1448d91cd8cac0ab321a9" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:22.642158 10461 raft_consensus.cc:3060] T d7df9d477cf1448d91cd8cac0ab321a9 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.642184 10596 raft_consensus.cc:3060] T d7df9d477cf1448d91cd8cac0ab321a9 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.643330 11018 leader_election.cc:290] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:22.645315 10461 raft_consensus.cc:2468] T d7df9d477cf1448d91cd8cac0ab321a9 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:22.645694 10254 leader_election.cc:304] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:22.645932 11018 raft_consensus.cc:2804] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:22.645917 10596 raft_consensus.cc:2468] T d7df9d477cf1448d91cd8cac0ab321a9 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:22.646087 11018 raft_consensus.cc:697] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:22.646735 11018 consensus_queue.cc:237] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
W20251025 01:55:22.651227 10398 connection.cc:537] server connection from 127.0.0.1:58386 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:22.651191 10532 connection.cc:537] server connection from 127.0.0.1:43676 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:22.651364 10649 connection.cc:537] client connection to 127.9.125.188:36641 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:22.652640 10046 connection.cc:537] server connection from 127.0.0.1:58666 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:22.652681 10256 connection.cc:537] server connection from 127.0.0.1:49362 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:22.653357 10254 connection.cc:537] client connection to 127.9.125.188:36641 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:22.653589 10531 connection.cc:537] client connection to 127.9.125.188:36641 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:22.654157 10395 connection.cc:537] client connection to 127.9.125.188:36641 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251025 01:55:22.719295 11018 raft_consensus.cc:493] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:22.719465 11018 raft_consensus.cc:515] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.720019 11018 leader_election.cc:290] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:22.720477 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0722d87a2814e589a20a0e4cb274c97" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:22.720512 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0722d87a2814e589a20a0e4cb274c97" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:22.720743 10596 raft_consensus.cc:2468] T a0722d87a2814e589a20a0e4cb274c97 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:22.720737 10461 raft_consensus.cc:2468] T a0722d87a2814e589a20a0e4cb274c97 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:22.721210 10255 leader_election.cc:304] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:22.721545 11018 raft_consensus.cc:2804] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:22.721676 11018 raft_consensus.cc:493] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:22.721772 11018 raft_consensus.cc:3060] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.723834 11018 raft_consensus.cc:515] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.724342 11018 leader_election.cc:290] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:22.724712 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0722d87a2814e589a20a0e4cb274c97" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:22.724766 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0722d87a2814e589a20a0e4cb274c97" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:22.724898 10596 raft_consensus.cc:3060] T a0722d87a2814e589a20a0e4cb274c97 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.724956 10461 raft_consensus.cc:3060] T a0722d87a2814e589a20a0e4cb274c97 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:22.726672 10596 raft_consensus.cc:2468] T a0722d87a2814e589a20a0e4cb274c97 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:22.726928 10461 raft_consensus.cc:2468] T a0722d87a2814e589a20a0e4cb274c97 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:22.727059 10255 leader_election.cc:304] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:22.727633 11018 raft_consensus.cc:2804] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:22.727778 11018 raft_consensus.cc:697] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:22.728240 11018 consensus_queue.cc:237] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:22.906215 11105 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 286b4c3025e24a7c8107763fa40d3ab1)
I20251025 01:55:22.906383 11105 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:22.907071 11105 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:22.908823 10046 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:22.909015 10044 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:22.909132 10044 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce; no voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:22.909540 11105 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 01:55:23.047430 11026 consensus_queue.cc:1048] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.052425 11111 consensus_queue.cc:1048] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.054857 10970 consensus_queue.cc:1048] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.061371 10970 consensus_queue.cc:1048] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.062558 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.190:35925
--webserver_interface=127.9.125.190
--webserver_port=45449
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251025 01:55:23.099506 11001 raft_consensus.cc:493] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:23.099742 11001 raft_consensus.cc:515] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:23.100572 11001 leader_election.cc:290] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:23.101140 10320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33df938c44154f41b333523a51b65e4e" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:23.101217 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33df938c44154f41b333523a51b65e4e" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:23.101338 10320 raft_consensus.cc:2468] T 33df938c44154f41b333523a51b65e4e P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 0.
I20251025 01:55:23.101468 10596 raft_consensus.cc:2468] T 33df938c44154f41b333523a51b65e4e P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 0.
I20251025 01:55:23.101914 10396 leader_election.cc:304] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2; no voters:
I20251025 01:55:23.102241 11001 raft_consensus.cc:2804] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:23.102370 11001 raft_consensus.cc:493] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:23.102474 11001 raft_consensus.cc:3060] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:23.105501 11001 raft_consensus.cc:515] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:23.106148 11001 leader_election.cc:290] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 election: Requested vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:23.106626 10320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33df938c44154f41b333523a51b65e4e" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
I20251025 01:55:23.106608 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33df938c44154f41b333523a51b65e4e" candidate_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:23.106791 10320 raft_consensus.cc:3060] T 33df938c44154f41b333523a51b65e4e P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:23.106815 10596 raft_consensus.cc:3060] T 33df938c44154f41b333523a51b65e4e P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:23.108804 10320 raft_consensus.cc:2468] T 33df938c44154f41b333523a51b65e4e P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 1.
I20251025 01:55:23.109166 10396 leader_election.cc:304] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:23.109407 11001 raft_consensus.cc:2804] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:23.109560 11001 raft_consensus.cc:697] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 LEADER]: Becoming Leader. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Running, Role: LEADER
I20251025 01:55:23.109897 10596 raft_consensus.cc:2468] T 33df938c44154f41b333523a51b65e4e P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f6e00cb735e4cb2bd1221cdcbc3c62c in term 1.
I20251025 01:55:23.110703 11001 consensus_queue.cc:237] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:23.158478 10970 consensus_queue.cc:1048] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.174340 10970 consensus_queue.cc:1048] T 3d37511aec3746cea09c1ba224cb14db P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.212153 10970 consensus_queue.cc:1048] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.217940 11018 consensus_queue.cc:1048] T 898ce3e93e56489f83a04fc46e7328de P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.391227 11001 consensus_queue.cc:1048] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.397188 11001 consensus_queue.cc:1048] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 01:55:23.421221 11121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:23.421707 11121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:23.421844 11121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 01:55:23.430539 11121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 01:55:23.430644 11121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:23.430776 11121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 01:55:23.430855 11121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 01:55:23.440969 11121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.190:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.125.190
--webserver_port=45449
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:23.442755 11121 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:23.444676 11121 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:23.452905 11137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:23.452886 11136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:23.453501 11139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:23.454329 11121 server_base.cc:1047] running on GCE node
I20251025 01:55:23.455046 11121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:23.456338 11121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:23.457516 11121 hybrid_clock.cc:648] HybridClock initialized: now 1761357323457452 us; error 58 us; skew 500 ppm
I20251025 01:55:23.457818 11121 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:23.459746 11121 webserver.cc:492] Webserver started at http://127.9.125.190:45449/ using document root <none> and password file <none>
I20251025 01:55:23.460384 11121 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:23.460485 11121 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:23.462208 11018 consensus_queue.cc:1048] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.462391 11144 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 286b4c3025e24a7c8107763fa40d3ab1)
I20251025 01:55:23.462570 11144 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:23.463359 11144 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:23.479394 10046 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:23.480356 10044 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:23.481123 10970 consensus_queue.cc:1048] T d7df9d477cf1448d91cd8cac0ab321a9 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.481216 10044 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce; no voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:23.481491 11144 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251025 01:55:23.484689 11121 fs_manager.cc:714] Time spent opening directory manager: real 0.022s user 0.012s sys 0.008s
I20251025 01:55:23.490150 11151 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:23.492436 11121 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20251025 01:55:23.492609 11121 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
uuid: "76559300dae44c0d8a17b583a8875b3f"
format_stamp: "Formatted at 2025-10-25 01:55:14 on dist-test-slave-mkd1"
I20251025 01:55:23.493342 11121 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:23.512429 11121 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:23.513173 11121 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:23.513429 11121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:23.532344 11121 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.190:35925
I20251025 01:55:23.532361 11202 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.190:35925 every 8 connection(s)
I20251025 01:55:23.534040 11121 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
I20251025 01:55:23.539530 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 11121
I20251025 01:55:23.541329 11203 sys_catalog.cc:263] Verifying existing consensus state
I20251025 01:55:23.546242 11203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Bootstrap starting.
I20251025 01:55:23.563665 11203 log.cc:826] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:23.620762 11203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=79 overwritten=0 applied=79 ignored=0} inserts{seen=62 ignored=0} mutations{seen=61 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251025 01:55:23.621292 11203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Bootstrap complete.
I20251025 01:55:23.624464 10970 consensus_queue.cc:1048] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.628094 11203 raft_consensus.cc:359] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:23.629014 11203 raft_consensus.cc:740] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76559300dae44c0d8a17b583a8875b3f, State: Initialized, Role: FOLLOWER
I20251025 01:55:23.630486 10970 consensus_queue.cc:1048] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.631081 11203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 79, Last appended: 3.79, Last appended by leader: 79, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:23.638142 10508 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:23.641659 11209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:23.641911 11209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:23.643154 11203 sys_catalog.cc:565] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: configured and running, proceeding with master startup.
I20251025 01:55:23.654727 10644 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:23.656239 10370 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:23.675947 11225 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:23.676043 11225 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: loading cluster ID for follower catalog manager: success
I20251025 01:55:23.680464 11225 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: acquiring CA information for follower catalog manager: success
I20251025 01:55:23.686040 11225 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 01:55:23.702180 11121 master_runner.cc:186] Error getting master registration for 127.9.125.188:36641: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111),
I20251025 01:55:23.705892 11121 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:23.773196 11018 consensus_queue.cc:1048] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.778905 11168 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" instance_seqno: 1761357317567829) as {username='slave'} at 127.9.125.131:43849; Asking this server to re-register.
I20251025 01:55:23.780083 10644 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:23.781399 10970 consensus_queue.cc:1048] T a0722d87a2814e589a20a0e4cb274c97 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:23.783629 11168 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" instance_seqno: 1761357316719619) as {username='slave'} at 127.9.125.129:55787; Asking this server to re-register.
I20251025 01:55:23.784329 11167 ts_manager.cc:194] Registered new tserver with Master: 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:23.785079 10370 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:23.786919 11167 ts_manager.cc:194] Registered new tserver with Master: c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:23.789043 11167 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" instance_seqno: 1761357317153916) as {username='slave'} at 127.9.125.130:44161; Asking this server to re-register.
I20251025 01:55:23.789763 10508 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:23.791225 11167 ts_manager.cc:194] Registered new tserver with Master: 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:24.092499 11001 consensus_queue.cc:1048] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:24.101402 11001 consensus_queue.cc:1048] T 33df938c44154f41b333523a51b65e4e P 1f6e00cb735e4cb2bd1221cdcbc3c62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:24.181674 11239 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:24.181906 11239 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:24.183632 11239 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:24.184152 11152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:24.184175 10070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 79 } } ignore_live_leader: false dest_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" is_pre_election: true
I20251025 01:55:24.184417 10070 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76559300dae44c0d8a17b583a8875b3f in current term 4: Already voted for candidate 286b4c3025e24a7c8107763fa40d3ab1 in this term.
W20251025 01:55:24.185866 11152 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:24.186008 11152 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:24.186312 11239 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 3 FOLLOWER]: Advancing to term 4
I20251025 01:55:24.189800 11239 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251025 01:55:24.444736 11241 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 286b4c3025e24a7c8107763fa40d3ab1)
I20251025 01:55:24.444900 11241 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:24.445605 11241 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:24.448221 10044 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:24.451975 11178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 85 } } ignore_live_leader: false dest_uuid: "76559300dae44c0d8a17b583a8875b3f" is_pre_election: true
I20251025 01:55:24.452390 11178 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ceaf0e1ce0f4c23b39a5bf4736de5ce in term 4.
I20251025 01:55:24.452848 10046 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f; no voters: 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:24.453111 11241 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251025 01:55:24.453209 11241 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Starting leader election (detected failure of leader 286b4c3025e24a7c8107763fa40d3ab1)
I20251025 01:55:24.453290 11241 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 4 FOLLOWER]: Advancing to term 5
I20251025 01:55:24.455184 11241 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:24.455775 11241 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 5 election: Requested vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:24.456251 11178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 85 } } ignore_live_leader: false dest_uuid: "76559300dae44c0d8a17b583a8875b3f"
I20251025 01:55:24.456419 11178 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 4 FOLLOWER]: Advancing to term 5
W20251025 01:55:24.457026 10044 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:24.458554 11178 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ceaf0e1ce0f4c23b39a5bf4736de5ce in term 5.
I20251025 01:55:24.458848 10046 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f; no voters: 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:24.459110 11241 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 5 FOLLOWER]: Leader election won for term 5
I20251025 01:55:24.459465 11241 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 5 LEADER]: Becoming Leader. State: Replica: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, State: Running, Role: LEADER
I20251025 01:55:24.459717 11241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 4.85, Last appended by leader: 85, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:24.460863 11246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce. Latest consensus state: current_term: 5 leader_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:24.461117 11246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:24.461576 11248 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 01:55:24.463691 11248 catalog_manager.cc:679] Loaded metadata for table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a]
I20251025 01:55:24.464025 11248 catalog_manager.cc:679] Loaded metadata for table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b]
I20251025 01:55:24.464341 11248 catalog_manager.cc:679] Loaded metadata for table default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc]
I20251025 01:55:24.464643 11248 catalog_manager.cc:679] Loaded metadata for table default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889]
I20251025 01:55:24.464980 11248 catalog_manager.cc:679] Loaded metadata for table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c]
I20251025 01:55:24.465348 11248 catalog_manager.cc:679] Loaded metadata for table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]
I20251025 01:55:24.465683 11248 catalog_manager.cc:679] Loaded metadata for table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980]
I20251025 01:55:24.466169 11248 catalog_manager.cc:679] Loaded metadata for table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]
I20251025 01:55:24.466483 11248 catalog_manager.cc:679] Loaded metadata for table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f]
I20251025 01:55:24.466779 11248 catalog_manager.cc:679] Loaded metadata for table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]
I20251025 01:55:24.467078 11248 catalog_manager.cc:679] Loaded metadata for table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]
I20251025 01:55:24.467552 11248 catalog_manager.cc:679] Loaded metadata for table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811]
I20251025 01:55:24.467845 11248 catalog_manager.cc:679] Loaded metadata for table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652]
I20251025 01:55:24.468156 11248 catalog_manager.cc:679] Loaded metadata for table default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737]
I20251025 01:55:24.468475 11248 catalog_manager.cc:679] Loaded metadata for table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2]
I20251025 01:55:24.468809 11248 catalog_manager.cc:679] Loaded metadata for table default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0]
I20251025 01:55:24.469120 11248 catalog_manager.cc:679] Loaded metadata for table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a]
I20251025 01:55:24.470574 11248 tablet_loader.cc:96] loaded metadata for tablet 0541393def0b4dcc9cf86038e01d1c21 (table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c])
I20251025 01:55:24.471019 11248 tablet_loader.cc:96] loaded metadata for tablet 174f2bfde6ef4ab4b90750c0f63fa9a1 (table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980])
I20251025 01:55:24.471467 11248 tablet_loader.cc:96] loaded metadata for tablet 33df938c44154f41b333523a51b65e4e (table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b])
I20251025 01:55:24.471738 11248 tablet_loader.cc:96] loaded metadata for tablet 3d37511aec3746cea09c1ba224cb14db (table default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0])
I20251025 01:55:24.472138 11248 tablet_loader.cc:96] loaded metadata for tablet 5906028cad8944bb97e90be8be25003f (table default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889])
I20251025 01:55:24.472399 11248 tablet_loader.cc:96] loaded metadata for tablet 5a2f9794447f4d7e8f6f548e4b26c8c6 (table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b])
I20251025 01:55:24.472767 11248 tablet_loader.cc:96] loaded metadata for tablet 87aaf87a6c284ca980e5fb6f896c5e4e (table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a])
I20251025 01:55:24.473008 11248 tablet_loader.cc:96] loaded metadata for tablet 898ce3e93e56489f83a04fc46e7328de (table default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc])
I20251025 01:55:24.474196 11248 tablet_loader.cc:96] loaded metadata for tablet a0722d87a2814e589a20a0e4cb274c97 (table default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737])
I20251025 01:55:24.474647 11248 tablet_loader.cc:96] loaded metadata for tablet a8bb9f9752b641499955e678b2ccc943 (table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811])
I20251025 01:55:24.474963 11248 tablet_loader.cc:96] loaded metadata for tablet b3578b8a1d8f412d9a8b2b968bb9c583 (table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2])
I20251025 01:55:24.475601 11248 tablet_loader.cc:96] loaded metadata for tablet cceaea14d4054de09cdaacb93f9b5582 (table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f])
I20251025 01:55:24.475898 11248 tablet_loader.cc:96] loaded metadata for tablet ce356b2a74154595bafdc91c27767b09 (table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a])
I20251025 01:55:24.476162 11248 tablet_loader.cc:96] loaded metadata for tablet d7df9d477cf1448d91cd8cac0ab321a9 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5])
I20251025 01:55:24.476578 11248 tablet_loader.cc:96] loaded metadata for tablet dcf81923a06d4750bbec550cff9bd91d (table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812])
I20251025 01:55:24.477013 11248 tablet_loader.cc:96] loaded metadata for tablet effc501d8ef24438a3fd7c30feb32f87 (table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433])
I20251025 01:55:24.477284 11248 tablet_loader.cc:96] loaded metadata for tablet f13a1ad12e2f4f2c95103cf8d76d6da6 (table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652])
I20251025 01:55:24.477711 11248 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 01:55:24.478876 11248 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:24.478943 11248 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 01:55:24.480322 11248 catalog_manager.cc:1514] Loading token signing keys...
I20251025 01:55:24.481441 11248 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Loaded TSK: 0
I20251025 01:55:24.481798 11248 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 01:55:24.482735 11248 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251025 01:55:24.483783 11248 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: 23
I20251025 01:55:24.501478 10055 master_service.cc:954] ReplaceTablet: received request to replace tablet 0541393def0b4dcc9cf86038e01d1c21 from {username='slave'} at 127.0.0.1:58658
I20251025 01:55:24.501559 10057 master_service.cc:954] ReplaceTablet: received request to replace tablet 898ce3e93e56489f83a04fc46e7328de from {username='slave'} at 127.0.0.1:58658
I20251025 01:55:24.502182 10053 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_19b5f67b2cfa4e62bc0d061b1c2c8ecf" } modify_external_catalogs: true
I20251025 01:55:24.502399 10053 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_19b5f67b2cfa4e62bc0d061b1c2c8ecf" } modify_external_catalogs: true
I20251025 01:55:24.502825 10052 master_service.cc:954] ReplaceTablet: received request to replace tablet 5906028cad8944bb97e90be8be25003f from {username='slave'} at 127.0.0.1:58658
I20251025 01:55:24.503086 10060 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_cf46809300e645fca4b99a2da934b149" } modify_external_catalogs: true
I20251025 01:55:24.503332 10060 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_cf46809300e645fca4b99a2da934b149" } modify_external_catalogs: true
I20251025 01:55:24.503906 10058 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_c18eb2691b2a4c5daf9746ee47a5e451" } new_table_name: "default.Table_7bca7380f82c44a4ac29fb6f6ed8a2fb" modify_external_catalogs: true
I20251025 01:55:24.504012 10054 master_service.cc:954] ReplaceTablet: received request to replace tablet ce356b2a74154595bafdc91c27767b09 from {username='slave'} at 127.0.0.1:58658
I20251025 01:55:24.504297 10056 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_827ca3c8c36543b29dac05ab520c552a" } new_table_name: "default.Table_6670e4ea42324ddc9e0c5a4317baf960" modify_external_catalogs: true
I20251025 01:55:24.505050 10051 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_44e4ccc4f2de47198a0599e2577108cf" } modify_external_catalogs: true
I20251025 01:55:24.505209 10051 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58658:
table { table_name: "default.Table_44e4ccc4f2de47198a0599e2577108cf" } modify_external_catalogs: true
I20251025 01:55:24.505357 10059 master_service.cc:954] ReplaceTablet: received request to replace tablet f13a1ad12e2f4f2c95103cf8d76d6da6 from {username='slave'} at 127.0.0.1:58658
F20251025 01:55:24.508285 10237 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 4 out of 13 received events: NotificationEvent(eventId=24, eventTime=1761357321, eventType=CREATE_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_a981c91dba574e1c8ccce55a41c4bd88, message={"server":"","servicePrincipal":"","db":"default","table":"table_a981c91dba574e1c8ccce55a41c4bd88","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a981c91dba574e1c8ccce55a41c4bd88\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357320},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_a981c91dba574e1c8ccce55a41c4bd88\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a981c91dba574e1c8ccce55a41c4bd88\",\"kudu.table_id\":\"269cfd64fbd94f0486843525917d9bb4\",\"transient_lastDdlTime\":\"1761357320\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_44e4ccc4f2de47198a0599e2577108cf, message={"server":"","servicePrincipal":"","db":"default","table":"table_44e4ccc4f2de47198a0599e2577108cf","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_44e4ccc4f2de47198a0599e2577108cf\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357319},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_44e4ccc4f2de47198a0599e2577108cf\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_44e4ccc4f2de47198a0599e2577108cf\",\"kudu.table_id\":\"58afab4104e84c868eeccda411b8452b\",\"transient_lastDdlTime\":\"1761357319\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251025 01:55:24.509501 11178 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Refusing update from remote peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 79. Preceding OpId from leader: term: 5 index: 87. (index mismatch)
I20251025 01:55:24.510193 11246 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 79, Time since last communication: 0.000s
@ 0x7f1293c86dcd google::LogMessage::Fail() at ??:0
W20251025 01:55:24.512069 10044 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce -> Peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Couldn't send request to peer 286b4c3025e24a7c8107763fa40d3ab1. Status: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f1293c8ab93 google::LogMessage::SendToLog() at ??:0
@ 0x7f1293c867cc google::LogMessage::Flush() at ??:0
@ 0x7f1293c86a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f12aa63fb64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251025 01:55:24.523550 11251 mvcc.cc:204] Tried to move back new op lower bound from 7214519592310509568 to 7214519591931486208. Current Snapshot: MvccSnapshot[applied={T|T < 7214519592192765952}]
@ 0x7f12aa63e411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251025 01:55:24.525815 11239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce. Latest consensus state: current_term: 5 leader_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:24.526005 11239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:24.525925 11253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:24.526274 11253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f12aa64608c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251025 01:55:24.529798 10055 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Sending DeleteTablet for 3 replicas of tablet 0541393def0b4dcc9cf86038e01d1c21
I20251025 01:55:24.529918 11253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:24.530077 11253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:24.530372 10052 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Sending DeleteTablet for 3 replicas of tablet 5906028cad8944bb97e90be8be25003f
I20251025 01:55:24.530903 10052 catalog_manager.cc:6719] ReplaceTablet: tablet 5906028cad8944bb97e90be8be25003f deleted and replaced by tablet 30ab28ea6ce847778e877ca0915448a6
I20251025 01:55:24.531805 10301 tablet_service.cc:1552] Processing DeleteTablet for tablet 5906028cad8944bb97e90be8be25003f with delete_type TABLET_DATA_DELETED (replaced by tablet 30ab28ea6ce847778e877ca0915448a6) from {username='slave'} at 127.0.0.1:49346
I20251025 01:55:24.531999 10574 tablet_service.cc:1552] Processing DeleteTablet for tablet 0541393def0b4dcc9cf86038e01d1c21 with delete_type TABLET_DATA_DELETED (replaced by tablet a600dcbd31104aee98146b72ee257034) from {username='slave'} at 127.0.0.1:43648
I20251025 01:55:24.532016 10440 tablet_service.cc:1552] Processing DeleteTablet for tablet 0541393def0b4dcc9cf86038e01d1c21 with delete_type TABLET_DATA_DELETED (replaced by tablet a600dcbd31104aee98146b72ee257034) from {username='slave'} at 127.0.0.1:58364
I20251025 01:55:24.532522 10300 tablet_service.cc:1552] Processing DeleteTablet for tablet 0541393def0b4dcc9cf86038e01d1c21 with delete_type TABLET_DATA_DELETED (replaced by tablet a600dcbd31104aee98146b72ee257034) from {username='slave'} at 127.0.0.1:49346
I20251025 01:55:24.533319 10578 tablet_service.cc:1552] Processing DeleteTablet for tablet 5906028cad8944bb97e90be8be25003f with delete_type TABLET_DATA_DELETED (replaced by tablet 30ab28ea6ce847778e877ca0915448a6) from {username='slave'} at 127.0.0.1:43648
I20251025 01:55:24.530393 10055 catalog_manager.cc:6719] ReplaceTablet: tablet 0541393def0b4dcc9cf86038e01d1c21 deleted and replaced by tablet a600dcbd31104aee98146b72ee257034
I20251025 01:55:24.533826 10439 tablet_service.cc:1552] Processing DeleteTablet for tablet 5906028cad8944bb97e90be8be25003f with delete_type TABLET_DATA_DELETED (replaced by tablet 30ab28ea6ce847778e877ca0915448a6) from {username='slave'} at 127.0.0.1:58364
@ 0x7f12aa645f09 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
I20251025 01:55:24.546103 11262 tablet_replica.cc:333] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:24.546351 11262 raft_consensus.cc:2243] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:24.546741 11262 raft_consensus.cc:2272] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:24.547428 11262 ts_tablet_manager.cc:1916] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:24.552129 11263 tablet_replica.cc:333] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:24.552357 11263 raft_consensus.cc:2243] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:24.552575 11263 raft_consensus.cc:2272] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:24.553203 11263 ts_tablet_manager.cc:1916] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f12969faee9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f129a0026db start_thread at ??:0
@ 0x7f1292b3771f clone at ??:0
I20251025 01:55:24.559500 11261 tablet_replica.cc:333] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:24.559773 11261 raft_consensus.cc:2243] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:24.560129 11261 raft_consensus.cc:2272] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:24.560695 11261 ts_tablet_manager.cc:1916] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:24.562075 11263 ts_tablet_manager.cc:1929] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:24.562170 11263 log.cc:1199] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/0541393def0b4dcc9cf86038e01d1c21
I20251025 01:55:24.562522 11263 ts_tablet_manager.cc:1950] T 0541393def0b4dcc9cf86038e01d1c21 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:24.566435 11261 ts_tablet_manager.cc:1929] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:24.566542 11261 log.cc:1199] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/5906028cad8944bb97e90be8be25003f
I20251025 01:55:24.566771 11262 ts_tablet_manager.cc:1929] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:24.566876 11262 log.cc:1199] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/0541393def0b4dcc9cf86038e01d1c21
I20251025 01:55:24.566972 11261 ts_tablet_manager.cc:1950] T 5906028cad8944bb97e90be8be25003f P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:24.567315 11262 ts_tablet_manager.cc:1950] T 0541393def0b4dcc9cf86038e01d1c21 P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:24.568019 11261 tablet_replica.cc:333] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:24.568215 11261 raft_consensus.cc:2243] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:24.568370 11261 raft_consensus.cc:2272] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:24.568424 11263 tablet_replica.cc:333] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:24.568611 11263 raft_consensus.cc:2243] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:24.568753 11263 raft_consensus.cc:2272] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:24.568853 11261 ts_tablet_manager.cc:1916] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:24.569237 11263 ts_tablet_manager.cc:1916] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:24.569789 11262 tablet_replica.cc:333] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:24.569978 11262 raft_consensus.cc:2243] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:24.570128 11262 raft_consensus.cc:2272] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:24.570621 11262 ts_tablet_manager.cc:1916] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:24.574445 11261 ts_tablet_manager.cc:1929] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:24.574549 11261 log.cc:1199] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/0541393def0b4dcc9cf86038e01d1c21
I20251025 01:55:24.574818 11263 ts_tablet_manager.cc:1929] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:24.574898 11261 ts_tablet_manager.cc:1950] T 0541393def0b4dcc9cf86038e01d1c21 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:24.574939 11263 log.cc:1199] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/5906028cad8944bb97e90be8be25003f
I20251025 01:55:24.575295 11263 ts_tablet_manager.cc:1950] T 5906028cad8944bb97e90be8be25003f P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:24.576496 11262 ts_tablet_manager.cc:1929] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:24.576617 11262 log.cc:1199] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/5906028cad8944bb97e90be8be25003f
I20251025 01:55:24.577008 11262 ts_tablet_manager.cc:1950] T 5906028cad8944bb97e90be8be25003f P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
W20251025 01:55:24.720582 10257 connection.cc:537] server connection from 127.0.0.1:49346 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:24.720858 10533 connection.cc:537] client connection to 127.9.125.189:45553 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:24.721015 10397 connection.cc:537] client connection to 127.9.125.189:45553 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:24.721302 10651 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.125.189:45553)
W20251025 01:55:24.721565 10651 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.125.189:45553)
W20251025 01:55:24.721719 10396 connection.cc:537] server connection from 127.0.0.1:58364 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:24.721812 10651 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.125.189:45553)
W20251025 01:55:24.722939 10534 connection.cc:537] server connection from 127.0.0.1:43648 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:24.723488 10651 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.125.189:45553)
I20251025 01:55:25.040709 11278 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:55:25.040871 11278 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:25.041437 11278 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:25.043067 11154 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:55:25.043207 11152 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:25.043376 11152 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:25.043594 11278 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 01:55:25.751953 11292 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:55:25.752128 11292 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:25.752724 11292 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:25.756304 11154 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:55:25.756470 11152 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:25.756588 11152 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:25.759907 11292 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 01:55:26.439512 11299 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:55:26.439672 11299 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:26.440184 11299 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:26.441778 11154 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:55:26.441924 11152 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:26.442060 11152 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:26.442278 11299 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251025 01:55:26.695042 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.188:36641
--webserver_interface=127.9.125.188
--webserver_port=33371
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251025 01:55:27.002159 11303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:27.002435 11303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:27.002513 11303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 01:55:27.010753 11303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 01:55:27.010823 11303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:27.010890 11303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 01:55:27.010934 11303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 01:55:27.019551 11303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.188:36641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.125.188
--webserver_port=33371
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:27.021080 11303 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:27.022706 11303 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:27.030366 11310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:27.030339 11311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:27.030602 11313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:27.031226 11303 server_base.cc:1047] running on GCE node
I20251025 01:55:27.031929 11303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:27.033171 11303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:27.034370 11303 hybrid_clock.cc:648] HybridClock initialized: now 1761357327034336 us; error 27 us; skew 500 ppm
I20251025 01:55:27.034646 11303 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:27.036520 11303 webserver.cc:492] Webserver started at http://127.9.125.188:33371/ using document root <none> and password file <none>
I20251025 01:55:27.037155 11303 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:27.037235 11303 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:27.042304 11303 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 01:55:27.045058 11319 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:27.046236 11303 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251025 01:55:27.046381 11303 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
uuid: "286b4c3025e24a7c8107763fa40d3ab1"
format_stamp: "Formatted at 2025-10-25 01:55:15 on dist-test-slave-mkd1"
I20251025 01:55:27.047127 11303 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:27.061110 11303 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:27.061714 11303 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:27.061908 11303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:27.081058 11303 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.188:36641
I20251025 01:55:27.081058 11371 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.188:36641 every 8 connection(s)
I20251025 01:55:27.082798 11303 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/info.pb
I20251025 01:55:27.089111 11372 sys_catalog.cc:263] Verifying existing consensus state
I20251025 01:55:27.091706 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 11303
I20251025 01:55:27.092281 11372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Bootstrap starting.
I20251025 01:55:27.109599 11372 log.cc:826] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:27.116609 10643 heartbeater.cc:344] Connected to a master server at 127.9.125.188:36641
I20251025 01:55:27.172550 11372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Bootstrap replayed 1/1 log segments. Stats: ops{read=86 overwritten=0 applied=85 ignored=0} inserts{seen=65 ignored=0} mutations{seen=70 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251025 01:55:27.173166 11372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Bootstrap complete.
I20251025 01:55:27.178165 11372 raft_consensus.cc:359] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:27.180505 11372 raft_consensus.cc:740] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286b4c3025e24a7c8107763fa40d3ab1, State: Initialized, Role: FOLLOWER
I20251025 01:55:27.181190 11372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 85, Last appended: 4.86, Last appended by leader: 86, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:27.181965 11378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:27.182235 11378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:27.182880 11372 sys_catalog.cc:565] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 01:55:27.198895 11392 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:27.199002 11392 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: loading cluster ID for follower catalog manager: success
I20251025 01:55:27.206972 11392 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: acquiring CA information for follower catalog manager: success
I20251025 01:55:27.211359 11392 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 01:55:27.221295 10507 heartbeater.cc:344] Connected to a master server at 127.9.125.188:36641
I20251025 01:55:27.224011 11336 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" instance_seqno: 1761357317153916) as {username='slave'} at 127.9.125.130:42499; Asking this server to re-register.
I20251025 01:55:27.224781 10507 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:27.226747 11336 ts_manager.cc:194] Registered new tserver with Master: 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:27.223742 11303 master_runner.cc:186] Error getting master registration for 127.9.125.189:45553: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111),
I20251025 01:55:27.229108 11303 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:27.401171 10369 heartbeater.cc:344] Connected to a master server at 127.9.125.188:36641
I20251025 01:55:27.401336 10369 heartbeater.cc:499] Master 127.9.125.188:36641 was elected leader, sending a full tablet report...
I20251025 01:55:27.403520 11336 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" instance_seqno: 1761357316719619) as {username='slave'} at 127.9.125.129:49549; Asking this server to re-register.
I20251025 01:55:27.404145 10369 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:27.405845 11336 ts_manager.cc:194] Registered new tserver with Master: c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:27.415872 11399 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:55:27.416028 11399 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:27.416597 11399 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:27.418921 11154 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:27.423218 11346 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 91 } } ignore_live_leader: false dest_uuid: "286b4c3025e24a7c8107763fa40d3ab1" is_pre_election: true
I20251025 01:55:27.423741 11346 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76559300dae44c0d8a17b583a8875b3f in term 4.
I20251025 01:55:27.424149 11152 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce
I20251025 01:55:27.424415 11399 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251025 01:55:27.424527 11399 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:55:27.424613 11399 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 5 FOLLOWER]: Advancing to term 6
I20251025 01:55:27.427526 11399 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:27.428210 11399 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 election: Requested vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:27.428701 11346 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 91 } } ignore_live_leader: false dest_uuid: "286b4c3025e24a7c8107763fa40d3ab1"
I20251025 01:55:27.428915 11346 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 4 FOLLOWER]: Advancing to term 6
W20251025 01:55:27.429651 11154 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:27.432533 11346 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76559300dae44c0d8a17b583a8875b3f in term 6.
I20251025 01:55:27.432873 11152 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce
I20251025 01:55:27.433120 11399 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 6 FOLLOWER]: Leader election won for term 6
I20251025 01:55:27.433501 11399 raft_consensus.cc:697] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 6 LEADER]: Becoming Leader. State: Replica: 76559300dae44c0d8a17b583a8875b3f, State: Running, Role: LEADER
I20251025 01:55:27.433949 11399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 5.91, Last appended by leader: 91, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:27.435806 11403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76559300dae44c0d8a17b583a8875b3f. Latest consensus state: current_term: 6 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:27.436084 11403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:27.436710 11405 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 01:55:27.439103 11405 catalog_manager.cc:679] Loaded metadata for table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a]
I20251025 01:55:27.439541 11405 catalog_manager.cc:679] Loaded metadata for table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b]
I20251025 01:55:27.439863 11405 catalog_manager.cc:679] Loaded metadata for table default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc]
I20251025 01:55:27.440176 11405 catalog_manager.cc:679] Loaded metadata for table default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889]
I20251025 01:55:27.440493 11405 catalog_manager.cc:679] Loaded metadata for table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c]
I20251025 01:55:27.440802 11405 catalog_manager.cc:679] Loaded metadata for table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]
I20251025 01:55:27.441084 11405 catalog_manager.cc:679] Loaded metadata for table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980]
I20251025 01:55:27.441526 11405 catalog_manager.cc:679] Loaded metadata for table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]
I20251025 01:55:27.441826 11405 catalog_manager.cc:679] Loaded metadata for table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f]
I20251025 01:55:27.442126 11405 catalog_manager.cc:679] Loaded metadata for table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]
I20251025 01:55:27.442418 11405 catalog_manager.cc:679] Loaded metadata for table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]
I20251025 01:55:27.442845 11405 catalog_manager.cc:679] Loaded metadata for table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811]
I20251025 01:55:27.443127 11405 catalog_manager.cc:679] Loaded metadata for table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652]
I20251025 01:55:27.443466 11405 catalog_manager.cc:679] Loaded metadata for table default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737]
I20251025 01:55:27.443753 11405 catalog_manager.cc:679] Loaded metadata for table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2]
I20251025 01:55:27.444078 11405 catalog_manager.cc:679] Loaded metadata for table default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0]
I20251025 01:55:27.444375 11405 catalog_manager.cc:679] Loaded metadata for table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a]
I20251025 01:55:27.446506 11405 tablet_loader.cc:96] loaded metadata for tablet 174f2bfde6ef4ab4b90750c0f63fa9a1 (table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980])
I20251025 01:55:27.446906 11405 tablet_loader.cc:96] loaded metadata for tablet 30ab28ea6ce847778e877ca0915448a6 (table default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889])
I20251025 01:55:27.447172 11405 tablet_loader.cc:96] loaded metadata for tablet 33df938c44154f41b333523a51b65e4e (table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b])
I20251025 01:55:27.447458 11405 tablet_loader.cc:96] loaded metadata for tablet 3a1411d1fa224fb9be8345d6daca1bcd (table default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc])
I20251025 01:55:27.447781 11405 tablet_loader.cc:96] loaded metadata for tablet 3d37511aec3746cea09c1ba224cb14db (table default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0])
I20251025 01:55:27.448423 11405 tablet_loader.cc:96] loaded metadata for tablet 5a2f9794447f4d7e8f6f548e4b26c8c6 (table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b])
I20251025 01:55:27.448813 11405 tablet_loader.cc:96] loaded metadata for tablet 87aaf87a6c284ca980e5fb6f896c5e4e (table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a])
I20251025 01:55:27.449599 11405 tablet_loader.cc:96] loaded metadata for tablet 937b8bb578ac4fdba8c1e800df5d6daa (table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a])
I20251025 01:55:27.450255 11405 tablet_loader.cc:96] loaded metadata for tablet a0722d87a2814e589a20a0e4cb274c97 (table default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737])
I20251025 01:55:27.450623 11405 tablet_loader.cc:96] loaded metadata for tablet a600dcbd31104aee98146b72ee257034 (table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c])
I20251025 01:55:27.450901 11405 tablet_loader.cc:96] loaded metadata for tablet a8bb9f9752b641499955e678b2ccc943 (table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811])
I20251025 01:55:27.451153 11405 tablet_loader.cc:96] loaded metadata for tablet b3578b8a1d8f412d9a8b2b968bb9c583 (table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2])
I20251025 01:55:27.451723 11405 tablet_loader.cc:96] loaded metadata for tablet cceaea14d4054de09cdaacb93f9b5582 (table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f])
I20251025 01:55:27.452198 11405 tablet_loader.cc:96] loaded metadata for tablet d7df9d477cf1448d91cd8cac0ab321a9 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5])
I20251025 01:55:27.452683 11405 tablet_loader.cc:96] loaded metadata for tablet dcf81923a06d4750bbec550cff9bd91d (table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812])
I20251025 01:55:27.453080 11405 tablet_loader.cc:96] loaded metadata for tablet eab40c1346c54c719060421fc54f8be4 (table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652])
I20251025 01:55:27.453356 11405 tablet_loader.cc:96] loaded metadata for tablet effc501d8ef24438a3fd7c30feb32f87 (table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433])
I20251025 01:55:27.453774 11405 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 01:55:27.454847 11405 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:27.454907 11405 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 01:55:27.456246 11405 catalog_manager.cc:1514] Loading token signing keys...
I20251025 01:55:27.457294 11405 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Loaded TSK: 0
I20251025 01:55:27.458278 11405 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 01:55:27.459385 11405 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251025 01:55:27.460541 11405 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: 23
I20251025 01:55:27.505386 11167 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48754:
table { table_name: "default.Table_8553cdea51804bfa861b6c10537b99c3" } new_table_name: "default.Table_89a70ae51ea04c43b2f1842ec3f87253" modify_external_catalogs: true
I20251025 01:55:27.506253 11168 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48754:
table { table_name: "default.Table_9187de27a70f48e08cc7284131fe543d" } modify_external_catalogs: true
I20251025 01:55:27.506536 11168 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48754:
table { table_name: "default.Table_9187de27a70f48e08cc7284131fe543d" } modify_external_catalogs: true
I20251025 01:55:27.507848 11161 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48754:
name: "default.Table_1a02719f7e0145b28ac84c3a43091e4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:27.515699 11159 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48754:
name: "default.Table_d93bab5cc39a448caf73455685fb742f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:27.511386 11166 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48754:
name: "default.Table_20c488ac6bf24db48ea2ab061d253e7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:27.508643 11160 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48754:
name: "default.Table_356101dc020f46c490a4bbc739447d43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:27.518165 11162 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48754:
name: "default.Table_50c0606a959945b3a1edd9df467a65dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:27.518422 11165 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48754:
name: "default.Table_a981c91dba574e1c8ccce55a41c4bd88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:27.518664 11163 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48754:
name: "default.Table_9328a3e2d017427796e74bbabc9f07bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251025 01:55:27.519390 11224 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 4 out of 13 received events: NotificationEvent(eventId=24, eventTime=1761357321, eventType=CREATE_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_a981c91dba574e1c8ccce55a41c4bd88, message={"server":"","servicePrincipal":"","db":"default","table":"table_a981c91dba574e1c8ccce55a41c4bd88","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a981c91dba574e1c8ccce55a41c4bd88\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357320},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_a981c91dba574e1c8ccce55a41c4bd88\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a981c91dba574e1c8ccce55a41c4bd88\",\"kudu.table_id\":\"269cfd64fbd94f0486843525917d9bb4\",\"transient_lastDdlTime\":\"1761357320\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_44e4ccc4f2de47198a0599e2577108cf, message={"server":"","servicePrincipal":"","db":"default","table":"table_44e4ccc4f2de47198a0599e2577108cf","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_44e4ccc4f2de47198a0599e2577108cf\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357319},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_44e4ccc4f2de47198a0599e2577108cf\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_44e4ccc4f2de47198a0599e2577108cf\",\"kudu.table_id\":\"58afab4104e84c868eeccda411b8452b\",\"transient_lastDdlTime\":\"1761357319\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251025 01:55:27.517949 11164 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48754:
name: "default.Table_ac8050cb49d54bc2a444d81cf8492c75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fdc5b15cdcd google::LogMessage::Fail() at ??:0
@ 0x7fdc5b160b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fdc5b15c7cc google::LogMessage::Flush() at ??:0
@ 0x7fdc5b15ca11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fdc71b15b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fdc71b14411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fdc71b1c08c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fdc71b1bf09 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7fdc5ded0ee9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdc614d86db start_thread at ??:0
@ 0x7fdc5a00d71f clone at ??:0
I20251025 01:55:27.622490 11336 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" instance_seqno: 1761357317567829) as {username='slave'} at 127.9.125.131:47485; Asking this server to re-register.
I20251025 01:55:27.623229 10643 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:27.624970 11336 ts_manager.cc:194] Registered new tserver with Master: 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
W20251025 01:55:27.720597 10533 connection.cc:537] client connection to 127.9.125.190:35925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:27.720780 10397 connection.cc:537] client connection to 127.9.125.190:35925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:27.720872 10651 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.125.190:35925)
W20251025 01:55:27.721462 10651 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.125.190:35925)
W20251025 01:55:27.721784 10651 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.125.190:35925)
W20251025 01:55:27.721969 10651 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.125.190:35925)
I20251025 01:55:28.035894 11420 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:28.036080 11420 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:28.037708 11420 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:28.038787 11322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:28.040413 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:28.040609 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:28.040707 11322 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:28.041018 11420 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251025 01:55:28.670091 11426 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:28.670287 11426 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:28.670857 11426 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:28.676154 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:28.676335 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:28.676431 11322 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:28.676656 11426 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251025 01:55:29.302225 11431 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:29.302390 11431 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:29.302935 11431 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:29.304814 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:29.305022 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:29.305102 11322 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:29.305327 11431 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251025 01:55:30.112560 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.190:35925
--webserver_interface=127.9.125.190
--webserver_port=45449
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251025 01:55:30.180789 11437 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:30.180971 11437 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:30.181584 11437 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:30.184005 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:30.184197 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:30.184288 11322 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:30.184499 11437 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251025 01:55:30.428535 11436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:30.428843 11436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:30.428941 11436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 01:55:30.436079 11436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 01:55:30.436168 11436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:30.436261 11436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 01:55:30.436316 11436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 01:55:30.444871 11436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.190:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.125.190
--webserver_port=45449
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:30.446331 11436 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:30.448001 11436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:30.455513 11445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:30.455569 11446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:30.456291 11448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:30.456580 11436 server_base.cc:1047] running on GCE node
I20251025 01:55:30.457212 11436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:30.458499 11436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:30.459688 11436 hybrid_clock.cc:648] HybridClock initialized: now 1761357330459635 us; error 36 us; skew 500 ppm
I20251025 01:55:30.459995 11436 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:30.461912 11436 webserver.cc:492] Webserver started at http://127.9.125.190:45449/ using document root <none> and password file <none>
I20251025 01:55:30.462548 11436 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:30.462635 11436 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:30.467234 11436 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20251025 01:55:30.469826 11454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:30.470901 11436 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 01:55:30.471033 11436 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
uuid: "76559300dae44c0d8a17b583a8875b3f"
format_stamp: "Formatted at 2025-10-25 01:55:14 on dist-test-slave-mkd1"
I20251025 01:55:30.471663 11436 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:30.486392 11436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:30.487051 11436 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:30.487320 11436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:30.504746 11436 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.190:35925
I20251025 01:55:30.504774 11505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.190:35925 every 8 connection(s)
I20251025 01:55:30.506038 11436 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
I20251025 01:55:30.509382 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 11436
I20251025 01:55:30.512820 11506 sys_catalog.cc:263] Verifying existing consensus state
I20251025 01:55:30.516294 11506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Bootstrap starting.
I20251025 01:55:30.532087 11506 log.cc:826] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:30.594152 11506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=92 overwritten=0 applied=91 ignored=0} inserts{seen=70 ignored=0} mutations{seen=75 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251025 01:55:30.594846 11506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Bootstrap complete.
I20251025 01:55:30.599813 11506 raft_consensus.cc:359] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:30.600653 11506 raft_consensus.cc:740] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76559300dae44c0d8a17b583a8875b3f, State: Initialized, Role: FOLLOWER
I20251025 01:55:30.601217 11506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 91, Last appended: 6.92, Last appended by leader: 92, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:30.601883 11510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:30.602115 11510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:30.602854 11506 sys_catalog.cc:565] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: configured and running, proceeding with master startup.
I20251025 01:55:30.614173 11524 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:30.614327 11524 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: loading cluster ID for follower catalog manager: success
I20251025 01:55:30.621753 11524 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: acquiring CA information for follower catalog manager: success
I20251025 01:55:30.625779 11524 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 01:55:30.627691 11436 master_runner.cc:186] Error getting master registration for 127.9.125.189:45553: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111),
I20251025 01:55:30.631507 11436 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:30.735975 10508 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:30.736094 10644 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:30.739277 11471 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" instance_seqno: 1761357317153916) as {username='slave'} at 127.9.125.130:59325; Asking this server to re-register.
I20251025 01:55:30.740228 10508 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:30.740260 11470 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" instance_seqno: 1761357317567829) as {username='slave'} at 127.9.125.131:34797; Asking this server to re-register.
I20251025 01:55:30.740835 10644 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:30.742300 11470 ts_manager.cc:194] Registered new tserver with Master: 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:30.742362 11471 ts_manager.cc:194] Registered new tserver with Master: 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:30.908550 10370 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:30.910149 11471 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" instance_seqno: 1761357316719619) as {username='slave'} at 127.9.125.129:36141; Asking this server to re-register.
I20251025 01:55:30.910741 10370 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:30.912146 11471 ts_manager.cc:194] Registered new tserver with Master: c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:31.210629 11529 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:31.210913 11529 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:31.212672 11529 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:31.213510 11457 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:55:31.215121 11457 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:31.218457 11346 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 92 } } ignore_live_leader: false dest_uuid: "286b4c3025e24a7c8107763fa40d3ab1" is_pre_election: true
I20251025 01:55:31.218750 11346 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76559300dae44c0d8a17b583a8875b3f in term 6.
I20251025 01:55:31.219205 11455 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce
I20251025 01:55:31.219554 11529 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251025 01:55:31.219689 11529 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:31.219786 11529 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 6 FOLLOWER]: Advancing to term 7
I20251025 01:55:31.223066 11529 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:31.223589 11529 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 7 election: Requested vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:31.224047 11346 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 92 } } ignore_live_leader: false dest_uuid: "286b4c3025e24a7c8107763fa40d3ab1"
I20251025 01:55:31.224278 11346 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 6 FOLLOWER]: Advancing to term 7
W20251025 01:55:31.224709 11457 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:31.226150 11346 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76559300dae44c0d8a17b583a8875b3f in term 7.
I20251025 01:55:31.226457 11455 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce
I20251025 01:55:31.226680 11529 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 7 FOLLOWER]: Leader election won for term 7
I20251025 01:55:31.227020 11529 raft_consensus.cc:697] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 7 LEADER]: Becoming Leader. State: Replica: 76559300dae44c0d8a17b583a8875b3f, State: Running, Role: LEADER
I20251025 01:55:31.227440 11529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 6.92, Last appended by leader: 92, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:31.229866 11534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76559300dae44c0d8a17b583a8875b3f. Latest consensus state: current_term: 7 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:31.230122 11534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:31.230634 11536 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 01:55:31.233181 11536 catalog_manager.cc:679] Loaded metadata for table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a]
I20251025 01:55:31.233691 11536 catalog_manager.cc:679] Loaded metadata for table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b]
I20251025 01:55:31.234114 11536 catalog_manager.cc:679] Loaded metadata for table default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc]
I20251025 01:55:31.234546 11536 catalog_manager.cc:679] Loaded metadata for table default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889]
I20251025 01:55:31.234962 11536 catalog_manager.cc:679] Loaded metadata for table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c]
I20251025 01:55:31.235409 11536 catalog_manager.cc:679] Loaded metadata for table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]
I20251025 01:55:31.235860 11536 catalog_manager.cc:679] Loaded metadata for table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980]
I20251025 01:55:31.236438 11536 catalog_manager.cc:679] Loaded metadata for table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]
I20251025 01:55:31.236716 11536 catalog_manager.cc:679] Loaded metadata for table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f]
I20251025 01:55:31.236994 11536 catalog_manager.cc:679] Loaded metadata for table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]
I20251025 01:55:31.237269 11536 catalog_manager.cc:679] Loaded metadata for table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]
I20251025 01:55:31.237697 11536 catalog_manager.cc:679] Loaded metadata for table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811]
I20251025 01:55:31.237978 11536 catalog_manager.cc:679] Loaded metadata for table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652]
I20251025 01:55:31.238265 11536 catalog_manager.cc:679] Loaded metadata for table default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737]
I20251025 01:55:31.238529 11536 catalog_manager.cc:679] Loaded metadata for table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2]
I20251025 01:55:31.238770 11536 catalog_manager.cc:679] Loaded metadata for table default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0]
I20251025 01:55:31.239008 11536 catalog_manager.cc:679] Loaded metadata for table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a]
I20251025 01:55:31.241166 11536 tablet_loader.cc:96] loaded metadata for tablet 174f2bfde6ef4ab4b90750c0f63fa9a1 (table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980])
I20251025 01:55:31.241556 11536 tablet_loader.cc:96] loaded metadata for tablet 30ab28ea6ce847778e877ca0915448a6 (table default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889])
I20251025 01:55:31.241789 11536 tablet_loader.cc:96] loaded metadata for tablet 33df938c44154f41b333523a51b65e4e (table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b])
I20251025 01:55:31.241976 11536 tablet_loader.cc:96] loaded metadata for tablet 3a1411d1fa224fb9be8345d6daca1bcd (table default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc])
I20251025 01:55:31.242192 11536 tablet_loader.cc:96] loaded metadata for tablet 3d37511aec3746cea09c1ba224cb14db (table default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0])
I20251025 01:55:31.242636 11536 tablet_loader.cc:96] loaded metadata for tablet 5a2f9794447f4d7e8f6f548e4b26c8c6 (table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b])
I20251025 01:55:31.242940 11536 tablet_loader.cc:96] loaded metadata for tablet 87aaf87a6c284ca980e5fb6f896c5e4e (table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a])
I20251025 01:55:31.243786 11536 tablet_loader.cc:96] loaded metadata for tablet 937b8bb578ac4fdba8c1e800df5d6daa (table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a])
I20251025 01:55:31.245201 11536 tablet_loader.cc:96] loaded metadata for tablet a0722d87a2814e589a20a0e4cb274c97 (table default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737])
I20251025 01:55:31.245702 11536 tablet_loader.cc:96] loaded metadata for tablet a600dcbd31104aee98146b72ee257034 (table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c])
I20251025 01:55:31.246004 11536 tablet_loader.cc:96] loaded metadata for tablet a8bb9f9752b641499955e678b2ccc943 (table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811])
I20251025 01:55:31.246248 11536 tablet_loader.cc:96] loaded metadata for tablet b3578b8a1d8f412d9a8b2b968bb9c583 (table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2])
I20251025 01:55:31.246794 11536 tablet_loader.cc:96] loaded metadata for tablet cceaea14d4054de09cdaacb93f9b5582 (table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f])
I20251025 01:55:31.247189 11536 tablet_loader.cc:96] loaded metadata for tablet d7df9d477cf1448d91cd8cac0ab321a9 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5])
I20251025 01:55:31.247642 11536 tablet_loader.cc:96] loaded metadata for tablet dcf81923a06d4750bbec550cff9bd91d (table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812])
I20251025 01:55:31.247988 11536 tablet_loader.cc:96] loaded metadata for tablet eab40c1346c54c719060421fc54f8be4 (table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652])
I20251025 01:55:31.248225 11536 tablet_loader.cc:96] loaded metadata for tablet effc501d8ef24438a3fd7c30feb32f87 (table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433])
I20251025 01:55:31.248610 11536 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 01:55:31.249734 11536 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:31.249797 11536 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 01:55:31.251107 11536 catalog_manager.cc:1514] Loading token signing keys...
I20251025 01:55:31.252200 11536 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Loaded TSK: 0
I20251025 01:55:31.252952 11536 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 01:55:31.254000 11536 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251025 01:55:31.255093 11536 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: 23
I20251025 01:55:31.276571 11470 master_service.cc:954] ReplaceTablet: received request to replace tablet 33df938c44154f41b333523a51b65e4e from {username='slave'} at 127.0.0.1:49482
I20251025 01:55:31.276688 11471 master_service.cc:954] ReplaceTablet: received request to replace tablet a0722d87a2814e589a20a0e4cb274c97 from {username='slave'} at 127.0.0.1:49482
I20251025 01:55:31.277410 11469 master_service.cc:954] ReplaceTablet: received request to replace tablet 898ce3e93e56489f83a04fc46e7328de from {username='slave'} at 127.0.0.1:49482
I20251025 01:55:31.277539 11466 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_cf46809300e645fca4b99a2da934b149" } modify_external_catalogs: true
I20251025 01:55:31.277843 11466 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_cf46809300e645fca4b99a2da934b149" } modify_external_catalogs: true
I20251025 01:55:31.279488 11465 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_44e4ccc4f2de47198a0599e2577108cf" } modify_external_catalogs: true
I20251025 01:55:31.279523 11469 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_ac8050cb49d54bc2a444d81cf8492c75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:31.279692 11465 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_44e4ccc4f2de47198a0599e2577108cf" } modify_external_catalogs: true
I20251025 01:55:31.279958 11468 master_service.cc:954] ReplaceTablet: received request to replace tablet ce356b2a74154595bafdc91c27767b09 from {username='slave'} at 127.0.0.1:49482
I20251025 01:55:31.280727 11467 master_service.cc:954] ReplaceTablet: received request to replace tablet f13a1ad12e2f4f2c95103cf8d76d6da6 from {username='slave'} at 127.0.0.1:49482
I20251025 01:55:31.281145 11462 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_9328a3e2d017427796e74bbabc9f07bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:31.281409 11467 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_20c488ac6bf24db48ea2ab061d253e7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:31.281958 11464 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_19b5f67b2cfa4e62bc0d061b1c2c8ecf" } modify_external_catalogs: true
I20251025 01:55:31.282258 11464 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_19b5f67b2cfa4e62bc0d061b1c2c8ecf" } modify_external_catalogs: true
I20251025 01:55:31.282920 11463 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_c18eb2691b2a4c5daf9746ee47a5e451" } new_table_name: "default.Table_7bca7380f82c44a4ac29fb6f6ed8a2fb" modify_external_catalogs: true
I20251025 01:55:31.280524 11468 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49482:
name: "default.Table_a981c91dba574e1c8ccce55a41c4bd88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251025 01:55:31.285495 11523 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 4 out of 13 received events: NotificationEvent(eventId=24, eventTime=1761357321, eventType=CREATE_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_a981c91dba574e1c8ccce55a41c4bd88, message={"server":"","servicePrincipal":"","db":"default","table":"table_a981c91dba574e1c8ccce55a41c4bd88","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a981c91dba574e1c8ccce55a41c4bd88\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357320},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_a981c91dba574e1c8ccce55a41c4bd88\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a981c91dba574e1c8ccce55a41c4bd88\",\"kudu.table_id\":\"269cfd64fbd94f0486843525917d9bb4\",\"transient_lastDdlTime\":\"1761357320\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_44e4ccc4f2de47198a0599e2577108cf, message={"server":"","servicePrincipal":"","db":"default","table":"table_44e4ccc4f2de47198a0599e2577108cf","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_44e4ccc4f2de47198a0599e2577108cf\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357319},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_44e4ccc4f2de47198a0599e2577108cf\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_44e4ccc4f2de47198a0599e2577108cf\",\"kudu.table_id\":\"58afab4104e84c868eeccda411b8452b\",\"transient_lastDdlTime\":\"1761357319\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251025 01:55:31.287953 11346 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Refusing update from remote peer 76559300dae44c0d8a17b583a8875b3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 86. Preceding OpId from leader: term: 7 index: 94. (index mismatch)
@ 0x7f8c33397dcd google::LogMessage::Fail() at ??:0
I20251025 01:55:31.288870 11539 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 85, Time since last communication: 0.000s
I20251025 01:55:31.289273 11539 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [LEADER]: Peer 286b4c3025e24a7c8107763fa40d3ab1 log is divergent from this leader: its last log entry 4.86 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 85
@ 0x7f8c3339bb93 google::LogMessage::SendToLog() at ??:0
W20251025 01:55:31.290318 11457 consensus_peers.cc:597] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f -> Peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Couldn't send request to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce. Status: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f8c333977cc google::LogMessage::Flush() at ??:0
@ 0x7f8c33397a11 google::LogMessage::~LogMessage() at ??:0
I20251025 01:55:31.292583 11346 pending_rounds.cc:85] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Aborting all ops after (but not including) 85
I20251025 01:55:31.292742 11346 pending_rounds.cc:107] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Aborting uncommitted WRITE_OP operation due to leader change: 4.86
@ 0x7f8c49d50b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f8c49d4f411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251025 01:55:31.308665 11546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:31.308940 11546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f8c49d5708c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251025 01:55:31.309260 11542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76559300dae44c0d8a17b583a8875b3f. Latest consensus state: current_term: 7 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:31.309520 11542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:31.309485 11538 mvcc.cc:204] Tried to move back new op lower bound from 7214519628941967360 to 7214519613173293056. Current Snapshot: MvccSnapshot[applied={T|T < 7214519628941967360}]
I20251025 01:55:31.310575 11542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:31.310773 11542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:31.310966 11542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:31.311118 11542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:31.311403 11543 mvcc.cc:204] Tried to move back new op lower bound from 7214519628945170432 to 7214519628711051264. Current Snapshot: MvccSnapshot[applied={T|T < 7214519628945170432 or (T in {7214519628945170432})}]
I20251025 01:55:31.311403 11529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:31.311673 11529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:31.312021 11534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:31.312189 11534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:31.312315 11471 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Sending DeleteTablet for 3 replicas of tablet a0722d87a2814e589a20a0e4cb274c97
I20251025 01:55:31.312317 11470 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Sending DeleteTablet for 3 replicas of tablet 33df938c44154f41b333523a51b65e4e
I20251025 01:55:31.314055 11470 catalog_manager.cc:6719] ReplaceTablet: tablet 33df938c44154f41b333523a51b65e4e deleted and replaced by tablet 0f9f9151de784d6d9e6ad0d75e76fede
I20251025 01:55:31.314496 11470 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_8553cdea51804bfa861b6c10537b99c3" } new_table_name: "default.Table_89a70ae51ea04c43b2f1842ec3f87253" modify_external_catalogs: true
@ 0x7f8c49d56f09 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
I20251025 01:55:31.318678 11471 catalog_manager.cc:6719] ReplaceTablet: tablet a0722d87a2814e589a20a0e4cb274c97 deleted and replaced by tablet 4e736e021c9d4855acbe0f56c1aa756f
I20251025 01:55:31.318994 11471 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_9187de27a70f48e08cc7284131fe543d" } modify_external_catalogs: true
I20251025 01:55:31.319233 11471 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49482:
table { table_name: "default.Table_9187de27a70f48e08cc7284131fe543d" } modify_external_catalogs: true
@ 0x7f8c3610bee9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8c397136db start_thread at ??:0
@ 0x7f8c3224871f clone at ??:0
W20251025 01:55:31.489230 10651 connection.cc:537] client connection to 127.9.125.190:35925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:31.489675 11554 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 01:55:31.315411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.9.125.130:43549)
1025 01:55:31.318815 (+ 3404us) server_negotiation.cc:207] Beginning negotiation
1025 01:55:31.318821 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 01:55:31.318839 (+ 18us) server_negotiation.cc:408] Connection header received
1025 01:55:31.318906 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 01:55:31.318911 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 01:55:31.318983 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 01:55:31.320037 (+ 1054us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 01:55:31.324591 (+ 4554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 01:55:31.326234 (+ 1643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 01:55:31.488801 (+162567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36452: BlockingRecv error: recv got EOF from 127.0.0.1:36452 (error 108)
Metrics: {"server-negotiator.queue_time_us":3287,"thread_start_us":6456,"threads_started":1}
W20251025 01:55:31.489733 11557 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 01:55:31.316279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.9.125.129:44581)
1025 01:55:31.333914 (+ 17635us) server_negotiation.cc:207] Beginning negotiation
1025 01:55:31.333920 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 01:55:31.333941 (+ 21us) server_negotiation.cc:408] Connection header received
1025 01:55:31.334017 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 01:55:31.334024 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 01:55:31.334101 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 01:55:31.334197 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 01:55:31.488771 (+154574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45870: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":17511,"thread_start_us":18116,"threads_started":1}
W20251025 01:55:31.489764 10651 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.125.190:35925)
W20251025 01:55:31.490137 10651 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.125.190:35925)
W20251025 01:55:31.490356 10256 connection.cc:537] client connection to 127.9.125.190:35925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:31.490437 10651 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.125.190:35925)
W20251025 01:55:31.490589 10651 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.125.190:35925)
W20251025 01:55:31.491007 11321 connection.cc:537] server connection from 127.0.0.1:40408 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251025 01:55:31.826205 11568 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 76559300dae44c0d8a17b583a8875b3f)
I20251025 01:55:31.826341 11568 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:31.826927 11568 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:31.828804 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:31.828977 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:31.829063 11322 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:31.829265 11568 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251025 01:55:32.230021 11303 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:55:32.230201 11303 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:32.425630 11575 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 76559300dae44c0d8a17b583a8875b3f)
I20251025 01:55:32.425873 11575 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:32.426518 11575 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:32.428267 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:32.428431 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:32.428532 11322 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:32.428773 11575 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251025 01:55:32.932760 11578 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 76559300dae44c0d8a17b583a8875b3f)
I20251025 01:55:32.932938 11578 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:32.933418 11578 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:32.935655 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:32.935892 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:32.935985 11322 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:32.936266 11578 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251025 01:55:33.772142 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.190:35925
--webserver_interface=127.9.125.190
--webserver_port=45449
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251025 01:55:34.079584 11583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:34.079916 11583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:34.080037 11583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 01:55:34.087726 11583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 01:55:34.087800 11583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:34.087863 11583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 01:55:34.087939 11583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 01:55:34.097105 11583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.190:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.125.190
--webserver_port=45449
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:34.098637 11583 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:34.100366 11583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:34.107463 11588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:34.107589 11589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:34.108477 11591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:34.109009 11583 server_base.cc:1047] running on GCE node
I20251025 01:55:34.109629 11583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:34.110806 11583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:34.112015 11583 hybrid_clock.cc:648] HybridClock initialized: now 1761357334111970 us; error 29 us; skew 500 ppm
I20251025 01:55:34.112310 11583 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:34.114214 11583 webserver.cc:492] Webserver started at http://127.9.125.190:45449/ using document root <none> and password file <none>
I20251025 01:55:34.114863 11583 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:34.114974 11583 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:34.119668 11583 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20251025 01:55:34.122124 11597 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:34.123383 11583 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251025 01:55:34.123530 11583 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
uuid: "76559300dae44c0d8a17b583a8875b3f"
format_stamp: "Formatted at 2025-10-25 01:55:14 on dist-test-slave-mkd1"
I20251025 01:55:34.124142 11583 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:34.140314 11583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:34.140964 11583 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:34.141177 11583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:34.166666 11583 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.190:35925
I20251025 01:55:34.166682 11648 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.190:35925 every 8 connection(s)
I20251025 01:55:34.168474 11583 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
I20251025 01:55:34.174970 11649 sys_catalog.cc:263] Verifying existing consensus state
I20251025 01:55:34.178165 11649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Bootstrap starting.
I20251025 01:55:34.178748 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 11583
I20251025 01:55:34.196889 11649 log.cc:826] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:34.257174 10644 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:34.258780 10508 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:34.258994 11654 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 76559300dae44c0d8a17b583a8875b3f)
I20251025 01:55:34.259181 11654 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:34.261355 11654 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:34.261598 11322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20251025 01:55:34.263000 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:34.270324 11649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=96 overwritten=0 applied=95 ignored=0} inserts{seen=72 ignored=0} mutations{seen=77 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251025 01:55:34.270967 11649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Bootstrap complete.
I20251025 01:55:34.276871 11649 raft_consensus.cc:359] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 7 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:34.279845 11649 raft_consensus.cc:740] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76559300dae44c0d8a17b583a8875b3f, State: Initialized, Role: FOLLOWER
I20251025 01:55:34.280536 11649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 95, Last appended: 7.96, Last appended by leader: 96, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:34.281315 11659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:34.281611 11659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:34.282383 11649 sys_catalog.cc:565] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: configured and running, proceeding with master startup.
I20251025 01:55:34.282462 11624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "286b4c3025e24a7c8107763fa40d3ab1" candidate_term: 8 candidate_status { last_received { term: 7 index: 96 } } ignore_live_leader: false dest_uuid: "76559300dae44c0d8a17b583a8875b3f" is_pre_election: true
I20251025 01:55:34.282953 11624 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 286b4c3025e24a7c8107763fa40d3ab1 in term 7.
I20251025 01:55:34.283556 11322 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce
I20251025 01:55:34.283973 11654 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251025 01:55:34.284093 11654 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 76559300dae44c0d8a17b583a8875b3f)
I20251025 01:55:34.284194 11654 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 7 FOLLOWER]: Advancing to term 8
I20251025 01:55:34.287456 11654 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:34.288244 11654 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 election: Requested vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
I20251025 01:55:34.289057 11624 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "286b4c3025e24a7c8107763fa40d3ab1" candidate_term: 8 candidate_status { last_received { term: 7 index: 96 } } ignore_live_leader: false dest_uuid: "76559300dae44c0d8a17b583a8875b3f"
I20251025 01:55:34.289270 11624 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 7 FOLLOWER]: Advancing to term 8
W20251025 01:55:34.290117 11322 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:34.293972 11624 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 286b4c3025e24a7c8107763fa40d3ab1 in term 8.
I20251025 01:55:34.294389 11322 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce
I20251025 01:55:34.294685 11654 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 8 FOLLOWER]: Leader election won for term 8
I20251025 01:55:34.295104 11654 raft_consensus.cc:697] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 8 LEADER]: Becoming Leader. State: Replica: 286b4c3025e24a7c8107763fa40d3ab1, State: Running, Role: LEADER
I20251025 01:55:34.295562 11654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 7.96, Last appended by leader: 96, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:34.301625 11673 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:34.301703 11673 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: loading cluster ID for follower catalog manager: success
I20251025 01:55:34.306849 11673 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: acquiring CA information for follower catalog manager: success
I20251025 01:55:34.310010 11673 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 01:55:34.314131 11676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286b4c3025e24a7c8107763fa40d3ab1. Latest consensus state: current_term: 8 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:34.314371 11676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:34.318455 11583 master_runner.cc:186] Error getting master registration for 127.9.125.189:45553: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111),
I20251025 01:55:34.323710 11583 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:34.503144 10370 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:34.505498 11613 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" instance_seqno: 1761357316719619) as {username='slave'} at 127.9.125.129:37995; Asking this server to re-register.
I20251025 01:55:34.506084 10370 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:34.508154 11613 ts_manager.cc:194] Registered new tserver with Master: c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:34.761833 11613 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" instance_seqno: 1761357317567829) as {username='slave'} at 127.9.125.131:59043; Asking this server to re-register.
I20251025 01:55:34.761971 11614 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" instance_seqno: 1761357317153916) as {username='slave'} at 127.9.125.130:56355; Asking this server to re-register.
I20251025 01:55:34.762501 10644 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:34.762593 10508 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:34.764024 11614 ts_manager.cc:194] Registered new tserver with Master: 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:34.764161 11613 ts_manager.cc:194] Registered new tserver with Master: 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:34.831364 11681 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:34.831574 11681 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:34.833420 11681 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:34.834259 11600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:55:34.835500 11600 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:34.839413 11345 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 9 candidate_status { last_received { term: 7 index: 96 } } ignore_live_leader: false dest_uuid: "286b4c3025e24a7c8107763fa40d3ab1" is_pre_election: true
I20251025 01:55:34.840022 11598 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:34.840315 11681 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251025 01:55:34.890592 11624 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Refusing update from remote peer 286b4c3025e24a7c8107763fa40d3ab1: Log matching property violated. Preceding OpId in replica: term: 7 index: 96. Preceding OpId from leader: term: 8 index: 97. (index mismatch)
I20251025 01:55:34.891335 11686 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 95, Time since last communication: 0.000s
I20251025 01:55:34.893783 11681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286b4c3025e24a7c8107763fa40d3ab1. Latest consensus state: current_term: 8 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:34.893972 11681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:34.901279 11681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:34.901468 11681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:34.902053 11688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "286b4c3025e24a7c8107763fa40d3ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:34.902465 11688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: LEADER
W20251025 01:55:34.902805 11322 consensus_peers.cc:597] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 -> Peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Couldn't send request to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce. Status: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 01:55:34.959496 11679 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 01:55:34.962215 11679 catalog_manager.cc:679] Loaded metadata for table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a]
I20251025 01:55:34.962611 11679 catalog_manager.cc:679] Loaded metadata for table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b]
I20251025 01:55:34.962937 11679 catalog_manager.cc:679] Loaded metadata for table default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc]
I20251025 01:55:34.963296 11679 catalog_manager.cc:679] Loaded metadata for table default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889]
I20251025 01:55:34.963696 11679 catalog_manager.cc:679] Loaded metadata for table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c]
I20251025 01:55:34.964093 11679 catalog_manager.cc:679] Loaded metadata for table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]
I20251025 01:55:34.964406 11679 catalog_manager.cc:679] Loaded metadata for table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980]
I20251025 01:55:34.964871 11679 catalog_manager.cc:679] Loaded metadata for table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]
I20251025 01:55:34.965174 11679 catalog_manager.cc:679] Loaded metadata for table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f]
I20251025 01:55:34.965477 11679 catalog_manager.cc:679] Loaded metadata for table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]
I20251025 01:55:34.965765 11679 catalog_manager.cc:679] Loaded metadata for table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]
I20251025 01:55:34.966204 11679 catalog_manager.cc:679] Loaded metadata for table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811]
I20251025 01:55:34.966517 11679 catalog_manager.cc:679] Loaded metadata for table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652]
I20251025 01:55:34.966825 11679 catalog_manager.cc:679] Loaded metadata for table default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737]
I20251025 01:55:34.967123 11679 catalog_manager.cc:679] Loaded metadata for table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2]
I20251025 01:55:34.967474 11679 catalog_manager.cc:679] Loaded metadata for table default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0]
I20251025 01:55:34.967819 11679 catalog_manager.cc:679] Loaded metadata for table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a]
I20251025 01:55:34.969862 11679 tablet_loader.cc:96] loaded metadata for tablet 0f9f9151de784d6d9e6ad0d75e76fede (table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b])
I20251025 01:55:34.970345 11679 tablet_loader.cc:96] loaded metadata for tablet 174f2bfde6ef4ab4b90750c0f63fa9a1 (table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980])
I20251025 01:55:34.970790 11679 tablet_loader.cc:96] loaded metadata for tablet 30ab28ea6ce847778e877ca0915448a6 (table default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889])
I20251025 01:55:34.971243 11679 tablet_loader.cc:96] loaded metadata for tablet 3a1411d1fa224fb9be8345d6daca1bcd (table default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc])
I20251025 01:55:34.971609 11679 tablet_loader.cc:96] loaded metadata for tablet 3d37511aec3746cea09c1ba224cb14db (table default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0])
I20251025 01:55:34.972048 11679 tablet_loader.cc:96] loaded metadata for tablet 4e736e021c9d4855acbe0f56c1aa756f (table default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737])
I20251025 01:55:34.972489 11679 tablet_loader.cc:96] loaded metadata for tablet 5a2f9794447f4d7e8f6f548e4b26c8c6 (table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b])
I20251025 01:55:34.972959 11679 tablet_loader.cc:96] loaded metadata for tablet 87aaf87a6c284ca980e5fb6f896c5e4e (table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a])
I20251025 01:55:34.973752 11679 tablet_loader.cc:96] loaded metadata for tablet 937b8bb578ac4fdba8c1e800df5d6daa (table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a])
I20251025 01:55:34.974671 11679 tablet_loader.cc:96] loaded metadata for tablet a600dcbd31104aee98146b72ee257034 (table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c])
I20251025 01:55:34.974936 11679 tablet_loader.cc:96] loaded metadata for tablet a8bb9f9752b641499955e678b2ccc943 (table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811])
I20251025 01:55:34.975199 11679 tablet_loader.cc:96] loaded metadata for tablet b3578b8a1d8f412d9a8b2b968bb9c583 (table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2])
I20251025 01:55:34.975775 11679 tablet_loader.cc:96] loaded metadata for tablet cceaea14d4054de09cdaacb93f9b5582 (table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f])
I20251025 01:55:34.976177 11679 tablet_loader.cc:96] loaded metadata for tablet d7df9d477cf1448d91cd8cac0ab321a9 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5])
I20251025 01:55:34.976662 11679 tablet_loader.cc:96] loaded metadata for tablet dcf81923a06d4750bbec550cff9bd91d (table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812])
I20251025 01:55:34.977155 11679 tablet_loader.cc:96] loaded metadata for tablet eab40c1346c54c719060421fc54f8be4 (table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652])
I20251025 01:55:34.977440 11679 tablet_loader.cc:96] loaded metadata for tablet effc501d8ef24438a3fd7c30feb32f87 (table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433])
I20251025 01:55:34.977846 11679 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 01:55:34.979005 11679 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:34.979087 11679 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 01:55:34.980471 11679 catalog_manager.cc:1514] Loading token signing keys...
I20251025 01:55:34.981580 11679 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Loaded TSK: 0
I20251025 01:55:34.982435 11679 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 01:55:34.983685 11679 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251025 01:55:34.984900 11679 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: 23
I20251025 01:55:35.042366 11335 master_service.cc:954] ReplaceTablet: received request to replace tablet a8bb9f9752b641499955e678b2ccc943 from {username='slave'} at 127.0.0.1:40394
I20251025 01:55:35.042351 11336 master_service.cc:954] ReplaceTablet: received request to replace tablet dcf81923a06d4750bbec550cff9bd91d from {username='slave'} at 127.0.0.1:40394
I20251025 01:55:35.045181 11334 master_service.cc:954] ReplaceTablet: received request to replace tablet 5a2f9794447f4d7e8f6f548e4b26c8c6 from {username='slave'} at 127.0.0.1:40394
I20251025 01:55:35.045982 11333 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40394:
table { table_name: "default.Table_827ca3c8c36543b29dac05ab520c552a" } new_table_name: "default.Table_6670e4ea42324ddc9e0c5a4317baf960" modify_external_catalogs: true
I20251025 01:55:35.046723 11332 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40394:
table { table_name: "default.Table_328c63794848430fb0f74a446d210d94" } new_table_name: "default.Table_68a9e5b0ffdd4bd081b95dca5aff0202" modify_external_catalogs: true
I20251025 01:55:35.047363 11331 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40394:
table { table_name: "default.Table_ac99b534589f45e6871ddfa3a1331b16" } new_table_name: "default.Table_5180205af52d470586f94f073f3537ca" modify_external_catalogs: true
I20251025 01:55:35.047879 11329 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40394:
name: "default.Table_b54f3b1b46af4568a75e8141684d8f97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:35.048162 11327 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40394:
name: "default.Table_d93bab5cc39a448caf73455685fb742f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:35.049224 11330 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40394:
table { table_name: "default.Table_986ff2b14daa4beebda43706d47dc98b" } modify_external_catalogs: true
I20251025 01:55:35.049502 11330 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40394:
table { table_name: "default.Table_986ff2b14daa4beebda43706d47dc98b" } modify_external_catalogs: true
I20251025 01:55:35.050011 11328 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40394:
name: "default.Table_9391ddd403f54e168c61fa879d7f8f6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:35.053237 11335 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Sending DeleteTablet for 3 replicas of tablet a8bb9f9752b641499955e678b2ccc943
I20251025 01:55:35.053427 11336 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Sending DeleteTablet for 3 replicas of tablet dcf81923a06d4750bbec550cff9bd91d
I20251025 01:55:35.054963 11335 catalog_manager.cc:6719] ReplaceTablet: tablet a8bb9f9752b641499955e678b2ccc943 deleted and replaced by tablet 0c92525ba189465fa7eff84f166da3d0
I20251025 01:55:35.055531 11335 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40394:
name: "default.Table_356101dc020f46c490a4bbc739447d43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:35.058413 11336 catalog_manager.cc:6719] ReplaceTablet: tablet dcf81923a06d4750bbec550cff9bd91d deleted and replaced by tablet ce8decc3a0fc4542bd94062e19c8fa55
I20251025 01:55:35.058738 11336 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40394:
name: "default.Table_1a02719f7e0145b28ac84c3a43091e4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:35.061205 11334 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Sending DeleteTablet for 3 replicas of tablet 5a2f9794447f4d7e8f6f548e4b26c8c6
I20251025 01:55:35.061707 11334 catalog_manager.cc:6719] ReplaceTablet: tablet 5a2f9794447f4d7e8f6f548e4b26c8c6 deleted and replaced by tablet fc9e67e3872b4990b0d0181eb96ad98c
I20251025 01:55:35.062012 11334 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40394:
table { table_name: "default.Table_cf46809300e645fca4b99a2da934b149" } modify_external_catalogs: true
I20251025 01:55:35.062299 11334 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40394:
table { table_name: "default.Table_cf46809300e645fca4b99a2da934b149" } modify_external_catalogs: true
F20251025 01:55:35.069877 11391 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 4 out of 13 received events: NotificationEvent(eventId=24, eventTime=1761357321, eventType=CREATE_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_a981c91dba574e1c8ccce55a41c4bd88, message={"server":"","servicePrincipal":"","db":"default","table":"table_a981c91dba574e1c8ccce55a41c4bd88","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a981c91dba574e1c8ccce55a41c4bd88\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357320},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_a981c91dba574e1c8ccce55a41c4bd88\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a981c91dba574e1c8ccce55a41c4bd88\",\"kudu.table_id\":\"269cfd64fbd94f0486843525917d9bb4\",\"transient_lastDdlTime\":\"1761357320\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_44e4ccc4f2de47198a0599e2577108cf, message={"server":"","servicePrincipal":"","db":"default","table":"table_44e4ccc4f2de47198a0599e2577108cf","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_44e4ccc4f2de47198a0599e2577108cf\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357319},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_44e4ccc4f2de47198a0599e2577108cf\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_44e4ccc4f2de47198a0599e2577108cf\",\"kudu.table_id\":\"58afab4104e84c868eeccda411b8452b\",\"transient_lastDdlTime\":\"1761357319\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fc8947d8dcd google::LogMessage::Fail() at ??:0
@ 0x7fc8947dcb93 google::LogMessage::SendToLog() at ??:0
I20251025 01:55:35.079969 10301 tablet_service.cc:1552] Processing DeleteTablet for tablet a8bb9f9752b641499955e678b2ccc943 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c92525ba189465fa7eff84f166da3d0) from {username='slave'} at 127.0.0.1:60320
I20251025 01:55:35.080334 10300 tablet_service.cc:1552] Processing DeleteTablet for tablet dcf81923a06d4750bbec550cff9bd91d with delete_type TABLET_DATA_DELETED (replaced by tablet ce8decc3a0fc4542bd94062e19c8fa55) from {username='slave'} at 127.0.0.1:60320
I20251025 01:55:35.080547 10298 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a2f9794447f4d7e8f6f548e4b26c8c6 with delete_type TABLET_DATA_DELETED (replaced by tablet fc9e67e3872b4990b0d0181eb96ad98c) from {username='slave'} at 127.0.0.1:60320
I20251025 01:55:35.085373 10298 tablet_service.cc:1505] Processing CreateTablet for tablet 0c92525ba189465fa7eff84f166da3d0 (DEFAULT_TABLE table=default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:35.085788 10298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c92525ba189465fa7eff84f166da3d0. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fc8947d87cc google::LogMessage::Flush() at ??:0
@ 0x7fc8947d8a11 google::LogMessage::~LogMessage() at ??:0
I20251025 01:55:35.091550 10300 tablet_service.cc:1505] Processing CreateTablet for tablet ce8decc3a0fc4542bd94062e19c8fa55 (DEFAULT_TABLE table=default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:35.091895 10300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce8decc3a0fc4542bd94062e19c8fa55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:35.097703 10440 tablet_service.cc:1552] Processing DeleteTablet for tablet a8bb9f9752b641499955e678b2ccc943 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c92525ba189465fa7eff84f166da3d0) from {username='slave'} at 127.0.0.1:42154
I20251025 01:55:35.097976 10297 tablet_service.cc:1505] Processing CreateTablet for tablet fc9e67e3872b4990b0d0181eb96ad98c (DEFAULT_TABLE table=default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7fc8ab191b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251025 01:55:35.098387 10297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9e67e3872b4990b0d0181eb96ad98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:35.098572 10439 tablet_service.cc:1552] Processing DeleteTablet for tablet dcf81923a06d4750bbec550cff9bd91d with delete_type TABLET_DATA_DELETED (replaced by tablet ce8decc3a0fc4542bd94062e19c8fa55) from {username='slave'} at 127.0.0.1:42154
I20251025 01:55:35.099069 10439 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a2f9794447f4d7e8f6f548e4b26c8c6 with delete_type TABLET_DATA_DELETED (replaced by tablet fc9e67e3872b4990b0d0181eb96ad98c) from {username='slave'} at 127.0.0.1:42154
I20251025 01:55:35.099819 10439 tablet_service.cc:1505] Processing CreateTablet for tablet 0c92525ba189465fa7eff84f166da3d0 (DEFAULT_TABLE table=default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:35.100214 10439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c92525ba189465fa7eff84f166da3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:35.100906 10574 tablet_service.cc:1552] Processing DeleteTablet for tablet dcf81923a06d4750bbec550cff9bd91d with delete_type TABLET_DATA_DELETED (replaced by tablet ce8decc3a0fc4542bd94062e19c8fa55) from {username='slave'} at 127.0.0.1:37674
I20251025 01:55:35.101065 10576 tablet_service.cc:1505] Processing CreateTablet for tablet 0c92525ba189465fa7eff84f166da3d0 (DEFAULT_TABLE table=default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:35.101253 10578 tablet_service.cc:1552] Processing DeleteTablet for tablet a8bb9f9752b641499955e678b2ccc943 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c92525ba189465fa7eff84f166da3d0) from {username='slave'} at 127.0.0.1:37674
I20251025 01:55:35.101470 10576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c92525ba189465fa7eff84f166da3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:35.101720 10577 tablet_service.cc:1505] Processing CreateTablet for tablet ce8decc3a0fc4542bd94062e19c8fa55 (DEFAULT_TABLE table=default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:35.102097 10577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce8decc3a0fc4542bd94062e19c8fa55. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fc8ab190411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251025 01:55:35.103700 10578 tablet_service.cc:1505] Processing CreateTablet for tablet fc9e67e3872b4990b0d0181eb96ad98c (DEFAULT_TABLE table=default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:35.104008 10578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9e67e3872b4990b0d0181eb96ad98c. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fc8ab19808c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251025 01:55:35.101476 10575 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a2f9794447f4d7e8f6f548e4b26c8c6 with delete_type TABLET_DATA_DELETED (replaced by tablet fc9e67e3872b4990b0d0181eb96ad98c) from {username='slave'} at 127.0.0.1:37674
@ 0x7fc8ab197f09 std::_Function_handler<>::_M_invoke() at ??:0
I20251025 01:55:35.111827 10437 tablet_service.cc:1505] Processing CreateTablet for tablet ce8decc3a0fc4542bd94062e19c8fa55 (DEFAULT_TABLE table=default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:35.112300 10437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce8decc3a0fc4542bd94062e19c8fa55. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0xa8867d std::function<>::operator()() at ??:?
I20251025 01:55:35.115623 10442 tablet_service.cc:1505] Processing CreateTablet for tablet fc9e67e3872b4990b0d0181eb96ad98c (DEFAULT_TABLE table=default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:35.116112 10442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9e67e3872b4990b0d0181eb96ad98c. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fc89754cee9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc89ab546db start_thread at ??:0
@ 0x7fc89368971f clone at ??:0
I20251025 01:55:35.134874 11709 tablet_replica.cc:333] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:35.135236 11709 raft_consensus.cc:2243] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:35.135536 11709 raft_consensus.cc:2272] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:35.136052 11711 tablet_bootstrap.cc:492] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:35.136345 11709 ts_tablet_manager.cc:1916] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:35.138402 11711 tablet_bootstrap.cc:654] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:35.140190 11707 tablet_bootstrap.cc:492] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:35.141206 11708 tablet_replica.cc:333] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:35.141424 11708 raft_consensus.cc:2243] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:35.141674 11708 raft_consensus.cc:2272] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:35.142235 11708 ts_tablet_manager.cc:1916] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:35.142779 11707 tablet_bootstrap.cc:654] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:35.144021 11705 tablet_replica.cc:333] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:35.144215 11705 raft_consensus.cc:2243] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:35.144269 11710 tablet_bootstrap.cc:492] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:35.144589 11705 raft_consensus.cc:2272] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:35.145129 11705 ts_tablet_manager.cc:1916] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:35.145944 11710 tablet_bootstrap.cc:654] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:35.148451 11710 tablet_bootstrap.cc:492] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:35.148581 11710 ts_tablet_manager.cc:1403] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251025 01:55:35.149362 11710 raft_consensus.cc:359] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.149554 11710 raft_consensus.cc:385] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:35.149628 11710 raft_consensus.cc:740] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:35.149866 11708 ts_tablet_manager.cc:1929] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 01:55:35.149813 11710 consensus_queue.cc:260] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.149976 11708 log.cc:1199] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/a8bb9f9752b641499955e678b2ccc943
I20251025 01:55:35.150336 11708 ts_tablet_manager.cc:1950] T a8bb9f9752b641499955e678b2ccc943 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:35.151420 11705 ts_tablet_manager.cc:1929] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 01:55:35.151537 11705 log.cc:1199] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/a8bb9f9752b641499955e678b2ccc943
I20251025 01:55:35.151942 11705 ts_tablet_manager.cc:1950] T a8bb9f9752b641499955e678b2ccc943 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:35.152902 11705 tablet_replica.cc:333] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:35.153077 11705 raft_consensus.cc:2243] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:35.153242 11705 raft_consensus.cc:2272] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:35.153677 11705 ts_tablet_manager.cc:1916] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:35.154361 11709 ts_tablet_manager.cc:1929] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:35.154456 11709 log.cc:1199] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/dcf81923a06d4750bbec550cff9bd91d
I20251025 01:55:35.154815 11709 ts_tablet_manager.cc:1950] T dcf81923a06d4750bbec550cff9bd91d P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:35.156713 11711 tablet_bootstrap.cc:492] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:35.156839 11711 ts_tablet_manager.cc:1403] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20251025 01:55:35.156992 11710 ts_tablet_manager.cc:1434] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251025 01:55:35.157263 11710 tablet_bootstrap.cc:492] T fc9e67e3872b4990b0d0181eb96ad98c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:35.159427 11705 ts_tablet_manager.cc:1929] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:35.159533 11705 log.cc:1199] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/dcf81923a06d4750bbec550cff9bd91d
I20251025 01:55:35.159919 11705 ts_tablet_manager.cc:1950] T dcf81923a06d4750bbec550cff9bd91d P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:35.159934 11710 tablet_bootstrap.cc:654] T fc9e67e3872b4990b0d0181eb96ad98c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:35.159983 11711 raft_consensus.cc:359] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.160154 11711 raft_consensus.cc:385] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:35.160229 11711 raft_consensus.cc:740] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:35.160468 11711 consensus_queue.cc:260] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.160902 11705 tablet_replica.cc:333] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
I20251025 01:55:35.161149 11705 raft_consensus.cc:2243] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:35.161518 11705 raft_consensus.cc:2272] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:35.162034 11705 ts_tablet_manager.cc:1916] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:35.162276 11707 tablet_bootstrap.cc:492] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:35.162405 11707 ts_tablet_manager.cc:1403] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.002s
I20251025 01:55:35.163393 11711 ts_tablet_manager.cc:1434] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251025 01:55:35.165620 11711 tablet_bootstrap.cc:492] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:35.167692 11705 ts_tablet_manager.cc:1929] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:35.167780 11705 log.cc:1199] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/5a2f9794447f4d7e8f6f548e4b26c8c6
I20251025 01:55:35.168118 11705 ts_tablet_manager.cc:1950] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:35.171938 11707 raft_consensus.cc:359] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.172115 11707 raft_consensus.cc:385] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:35.172207 11707 raft_consensus.cc:740] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:35.172442 11707 consensus_queue.cc:260] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.172859 11707 ts_tablet_manager.cc:1434] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251025 01:55:35.173135 11707 tablet_bootstrap.cc:492] T fc9e67e3872b4990b0d0181eb96ad98c P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:35.174695 11707 tablet_bootstrap.cc:654] T fc9e67e3872b4990b0d0181eb96ad98c P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:35.176261 11710 tablet_bootstrap.cc:492] T fc9e67e3872b4990b0d0181eb96ad98c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:35.176391 11710 ts_tablet_manager.cc:1403] T fc9e67e3872b4990b0d0181eb96ad98c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20251025 01:55:35.177064 11710 raft_consensus.cc:359] T fc9e67e3872b4990b0d0181eb96ad98c P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.177265 11710 raft_consensus.cc:385] T fc9e67e3872b4990b0d0181eb96ad98c P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:35.177364 11710 raft_consensus.cc:740] T fc9e67e3872b4990b0d0181eb96ad98c P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:35.177580 11710 consensus_queue.cc:260] T fc9e67e3872b4990b0d0181eb96ad98c P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.178025 11710 ts_tablet_manager.cc:1434] T fc9e67e3872b4990b0d0181eb96ad98c P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 01:55:35.178642 11709 tablet_replica.cc:333] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:35.178831 11709 raft_consensus.cc:2243] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:35.178988 11709 raft_consensus.cc:2272] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:35.179526 11709 ts_tablet_manager.cc:1916] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:35.180545 11711 tablet_bootstrap.cc:654] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:35.182798 11710 tablet_bootstrap.cc:492] T ce8decc3a0fc4542bd94062e19c8fa55 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Bootstrap starting.
I20251025 01:55:35.183380 11708 tablet_replica.cc:333] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:35.183666 11708 raft_consensus.cc:2243] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:35.184039 11708 raft_consensus.cc:2272] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:35.184686 11708 ts_tablet_manager.cc:1916] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:35.185478 11709 ts_tablet_manager.cc:1929] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 01:55:35.185607 11709 log.cc:1199] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/a8bb9f9752b641499955e678b2ccc943
I20251025 01:55:35.185864 11710 tablet_bootstrap.cc:654] T ce8decc3a0fc4542bd94062e19c8fa55 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:35.186010 11709 ts_tablet_manager.cc:1950] T a8bb9f9752b641499955e678b2ccc943 P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:35.186889 11709 tablet_replica.cc:333] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2: stopping tablet replica
I20251025 01:55:35.187041 11709 raft_consensus.cc:2243] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:35.187188 11709 raft_consensus.cc:2272] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:35.187683 11709 ts_tablet_manager.cc:1916] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:35.193531 11709 ts_tablet_manager.cc:1929] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:35.193635 11709 log.cc:1199] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-2/wal/wals/5a2f9794447f4d7e8f6f548e4b26c8c6
I20251025 01:55:35.194000 11709 ts_tablet_manager.cc:1950] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 90109d2fdb464942b20891b5856f8ac2: Deleting consensus metadata
I20251025 01:55:35.195561 11707 tablet_bootstrap.cc:492] T fc9e67e3872b4990b0d0181eb96ad98c P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:35.195705 11710 tablet_bootstrap.cc:492] T ce8decc3a0fc4542bd94062e19c8fa55 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: No bootstrap required, opened a new log
I20251025 01:55:35.195706 11707 ts_tablet_manager.cc:1403] T fc9e67e3872b4990b0d0181eb96ad98c P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.001s
I20251025 01:55:35.195829 11710 ts_tablet_manager.cc:1403] T ce8decc3a0fc4542bd94062e19c8fa55 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20251025 01:55:35.196391 11707 raft_consensus.cc:359] T fc9e67e3872b4990b0d0181eb96ad98c P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.196563 11707 raft_consensus.cc:385] T fc9e67e3872b4990b0d0181eb96ad98c P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:35.196630 11707 raft_consensus.cc:740] T fc9e67e3872b4990b0d0181eb96ad98c P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:35.196573 11710 raft_consensus.cc:359] T ce8decc3a0fc4542bd94062e19c8fa55 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.196754 11710 raft_consensus.cc:385] T ce8decc3a0fc4542bd94062e19c8fa55 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:35.196827 11710 raft_consensus.cc:740] T ce8decc3a0fc4542bd94062e19c8fa55 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6e00cb735e4cb2bd1221cdcbc3c62c, State: Initialized, Role: FOLLOWER
I20251025 01:55:35.196897 11707 consensus_queue.cc:260] T fc9e67e3872b4990b0d0181eb96ad98c P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.197053 11710 consensus_queue.cc:260] T ce8decc3a0fc4542bd94062e19c8fa55 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.197485 11711 tablet_bootstrap.cc:492] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:35.197510 11710 ts_tablet_manager.cc:1434] T ce8decc3a0fc4542bd94062e19c8fa55 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 01:55:35.197613 11711 ts_tablet_manager.cc:1403] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20251025 01:55:35.198776 11711 raft_consensus.cc:359] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.198961 11711 raft_consensus.cc:385] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:35.199044 11711 raft_consensus.cc:740] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:35.199369 11711 consensus_queue.cc:260] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.199505 11707 ts_tablet_manager.cc:1434] T fc9e67e3872b4990b0d0181eb96ad98c P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251025 01:55:35.200525 11707 tablet_bootstrap.cc:492] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:35.202442 11707 tablet_bootstrap.cc:654] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:35.203083 11711 ts_tablet_manager.cc:1434] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 01:55:35.205020 11711 tablet_bootstrap.cc:492] T ce8decc3a0fc4542bd94062e19c8fa55 P 90109d2fdb464942b20891b5856f8ac2: Bootstrap starting.
I20251025 01:55:35.205210 11707 tablet_bootstrap.cc:492] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:35.205322 11707 ts_tablet_manager.cc:1403] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20251025 01:55:35.205910 11707 raft_consensus.cc:359] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.206061 11707 raft_consensus.cc:385] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:35.206138 11707 raft_consensus.cc:740] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:35.206317 11707 consensus_queue.cc:260] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.206616 11711 tablet_bootstrap.cc:654] T ce8decc3a0fc4542bd94062e19c8fa55 P 90109d2fdb464942b20891b5856f8ac2: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:35.206763 11707 ts_tablet_manager.cc:1434] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 01:55:35.209261 11708 ts_tablet_manager.cc:1929] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:35.209370 11708 log.cc:1199] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/dcf81923a06d4750bbec550cff9bd91d
I20251025 01:55:35.209718 11708 ts_tablet_manager.cc:1950] T dcf81923a06d4750bbec550cff9bd91d P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:35.210542 11708 tablet_replica.cc:333] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:35.210956 11708 raft_consensus.cc:2243] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:35.211153 11708 raft_consensus.cc:2272] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:35.211694 11708 ts_tablet_manager.cc:1916] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:35.218125 11708 ts_tablet_manager.cc:1929] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:35.218254 11708 log.cc:1199] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/5a2f9794447f4d7e8f6f548e4b26c8c6
I20251025 01:55:35.218649 11708 ts_tablet_manager.cc:1950] T 5a2f9794447f4d7e8f6f548e4b26c8c6 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:35.222223 11711 tablet_bootstrap.cc:492] T ce8decc3a0fc4542bd94062e19c8fa55 P 90109d2fdb464942b20891b5856f8ac2: No bootstrap required, opened a new log
I20251025 01:55:35.222347 11711 ts_tablet_manager.cc:1403] T ce8decc3a0fc4542bd94062e19c8fa55 P 90109d2fdb464942b20891b5856f8ac2: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.002s
I20251025 01:55:35.223098 11711 raft_consensus.cc:359] T ce8decc3a0fc4542bd94062e19c8fa55 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.223369 11711 raft_consensus.cc:385] T ce8decc3a0fc4542bd94062e19c8fa55 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:35.223467 11711 raft_consensus.cc:740] T ce8decc3a0fc4542bd94062e19c8fa55 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Initialized, Role: FOLLOWER
I20251025 01:55:35.223762 11711 consensus_queue.cc:260] T ce8decc3a0fc4542bd94062e19c8fa55 P 90109d2fdb464942b20891b5856f8ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.225703 11711 ts_tablet_manager.cc:1434] T ce8decc3a0fc4542bd94062e19c8fa55 P 90109d2fdb464942b20891b5856f8ac2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 01:55:35.285682 11680 raft_consensus.cc:493] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:35.285872 11680 raft_consensus.cc:515] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.286754 11680 leader_election.cc:290] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:35.287362 10320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc9e67e3872b4990b0d0181eb96ad98c" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" is_pre_election: true
I20251025 01:55:35.287405 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc9e67e3872b4990b0d0181eb96ad98c" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:35.287616 10320 raft_consensus.cc:2468] T fc9e67e3872b4990b0d0181eb96ad98c P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:35.287688 10461 raft_consensus.cc:2468] T fc9e67e3872b4990b0d0181eb96ad98c P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 0.
I20251025 01:55:35.288100 10532 leader_election.cc:304] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:35.288425 11680 raft_consensus.cc:2804] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:35.288565 11680 raft_consensus.cc:493] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:35.288655 11680 raft_consensus.cc:3060] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:35.291594 11680 raft_consensus.cc:515] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.292299 11680 leader_election.cc:290] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:35.292789 10320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc9e67e3872b4990b0d0181eb96ad98c" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b"
I20251025 01:55:35.292799 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc9e67e3872b4990b0d0181eb96ad98c" candidate_uuid: "90109d2fdb464942b20891b5856f8ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:35.292985 10461 raft_consensus.cc:3060] T fc9e67e3872b4990b0d0181eb96ad98c P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:35.293005 10320 raft_consensus.cc:3060] T fc9e67e3872b4990b0d0181eb96ad98c P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:35.294896 10320 raft_consensus.cc:2468] T fc9e67e3872b4990b0d0181eb96ad98c P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:35.295332 10532 leader_election.cc:304] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:35.295579 11680 raft_consensus.cc:2804] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:35.295790 10461 raft_consensus.cc:2468] T fc9e67e3872b4990b0d0181eb96ad98c P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90109d2fdb464942b20891b5856f8ac2 in term 1.
I20251025 01:55:35.296010 11680 raft_consensus.cc:697] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 90109d2fdb464942b20891b5856f8ac2, State: Running, Role: LEADER
I20251025 01:55:35.296684 11680 consensus_queue.cc:237] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.333788 11558 raft_consensus.cc:493] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:35.333986 11558 raft_consensus.cc:515] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.335024 11558 leader_election.cc:290] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:35.335386 10461 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce8decc3a0fc4542bd94062e19c8fa55" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:35.335794 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce8decc3a0fc4542bd94062e19c8fa55" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:35.335888 10461 raft_consensus.cc:2468] T ce8decc3a0fc4542bd94062e19c8fa55 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:35.336056 10596 raft_consensus.cc:2468] T ce8decc3a0fc4542bd94062e19c8fa55 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:35.336532 10255 leader_election.cc:304] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:35.336836 11732 raft_consensus.cc:2804] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:35.336928 11732 raft_consensus.cc:493] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:35.337030 11732 raft_consensus.cc:3060] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:35.338938 11732 raft_consensus.cc:515] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.339556 11732 leader_election.cc:290] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:35.339934 10462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce8decc3a0fc4542bd94062e19c8fa55" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:35.339996 10596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce8decc3a0fc4542bd94062e19c8fa55" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:35.340131 10462 raft_consensus.cc:3060] T ce8decc3a0fc4542bd94062e19c8fa55 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:35.340170 10596 raft_consensus.cc:3060] T ce8decc3a0fc4542bd94062e19c8fa55 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
W20251025 01:55:35.342844 10395 connection.cc:537] server connection from 127.0.0.1:42154 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:35.342808 10255 connection.cc:537] server connection from 127.0.0.1:60320 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251025 01:55:35.342923 10596 raft_consensus.cc:2468] T ce8decc3a0fc4542bd94062e19c8fa55 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:35.342944 10462 raft_consensus.cc:2468] T ce8decc3a0fc4542bd94062e19c8fa55 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
W20251025 01:55:35.343084 10533 connection.cc:537] server connection from 127.0.0.1:37674 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251025 01:55:35.343611 10255 leader_election.cc:304] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
W20251025 01:55:35.343879 10254 connection.cc:537] client connection to 127.9.125.188:36641 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251025 01:55:35.344290 11732 raft_consensus.cc:2804] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:35.344446 11732 raft_consensus.cc:697] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:35.344944 11732 consensus_queue.cc:237] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
W20251025 01:55:35.345319 10649 connection.cc:537] client connection to 127.9.125.188:36641 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:35.346326 10649 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.125.188:36641)
W20251025 01:55:35.346493 10395 connection.cc:537] client connection to 127.9.125.188:36641 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:35.346984 10649 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.125.188:36641)
W20251025 01:55:35.347196 10649 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.125.188:36641)
W20251025 01:55:35.347483 10649 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.125.188:36641)
W20251025 01:55:35.349350 10531 connection.cc:537] client connection to 127.9.125.188:36641 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:35.349802 11572 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 01:55:35.346857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.125.188:36641 (local address 127.9.125.129:48859)
1025 01:55:35.347101 (+ 244us) negotiation.cc:107] Waiting for socket to connect
1025 01:55:35.347120 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1025 01:55:35.347233 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 01:55:35.349605 (+ 2372us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126}
W20251025 01:55:35.350977 11566 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 01:55:35.349482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.125.188:36641 (local address 127.9.125.130:52753)
1025 01:55:35.350127 (+ 645us) negotiation.cc:107] Waiting for socket to connect
1025 01:55:35.350144 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 01:55:35.350241 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 01:55:35.350835 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":544}
I20251025 01:55:35.581899 11732 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:35.582098 11732 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.582749 11732 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:35.583186 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c92525ba189465fa7eff84f166da3d0" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2" is_pre_election: true
I20251025 01:55:35.583214 10462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c92525ba189465fa7eff84f166da3d0" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" is_pre_election: true
I20251025 01:55:35.583429 10598 raft_consensus.cc:2468] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:35.583487 10462 raft_consensus.cc:2468] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 0.
I20251025 01:55:35.583808 10255 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:35.584100 11732 raft_consensus.cc:2804] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 01:55:35.584210 11732 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:35.584322 11732 raft_consensus.cc:3060] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:35.586277 11732 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.586828 11732 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Requested vote from peers 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549), 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:35.587263 10462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c92525ba189465fa7eff84f166da3d0" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c"
I20251025 01:55:35.587322 10598 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c92525ba189465fa7eff84f166da3d0" candidate_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90109d2fdb464942b20891b5856f8ac2"
I20251025 01:55:35.587479 10462 raft_consensus.cc:3060] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:35.587487 10598 raft_consensus.cc:3060] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 01:55:35.590353 10462 raft_consensus.cc:2468] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:35.590380 10598 raft_consensus.cc:2468] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e4f9f2253e416e8b31ea86ead23a3b in term 1.
I20251025 01:55:35.590699 10254 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b; no voters:
I20251025 01:55:35.590953 11732 raft_consensus.cc:2804] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 01:55:35.591099 11732 raft_consensus.cc:697] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Becoming Leader. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Running, Role: LEADER
I20251025 01:55:35.591390 11732 consensus_queue.cc:237] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:35.596688 11737 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 286b4c3025e24a7c8107763fa40d3ab1)
I20251025 01:55:35.596845 11737 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:35.597477 11737 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:35.600682 11600 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:55:35.600914 11598 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:35.601054 11598 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:35.601317 11737 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251025 01:55:36.152091 11732 consensus_queue.cc:1048] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:36.158313 11745 consensus_queue.cc:1048] T ce8decc3a0fc4542bd94062e19c8fa55 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:36.238313 11753 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 286b4c3025e24a7c8107763fa40d3ab1)
I20251025 01:55:36.238529 11753 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:36.239393 11753 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:36.241561 11598 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
W20251025 01:55:36.241720 11600 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:36.241879 11600 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:36.242309 11753 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251025 01:55:36.274415 11749 consensus_queue.cc:1048] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:36.281524 11758 consensus_queue.cc:1048] T fc9e67e3872b4990b0d0181eb96ad98c P 90109d2fdb464942b20891b5856f8ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 01:55:36.358057 9718 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu pid 11583
I20251025 01:55:36.364735 11644 generic_service.cc:196] Checking for leaks (request via RPC)
I20251025 01:55:36.470870 11748 consensus_queue.cc:1048] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:36.476528 11745 consensus_queue.cc:1048] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 01:55:36.484647 9718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu with pid 11583
W20251025 01:55:36.506248 10397 connection.cc:537] client connection to 127.9.125.190:35925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251025 01:55:37.485175 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.190:35925
--webserver_interface=127.9.125.190
--webserver_port=45449
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251025 01:55:37.790983 11779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:37.791276 11779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:37.791424 11779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 01:55:37.799036 11779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 01:55:37.799120 11779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:37.799206 11779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 01:55:37.799310 11779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 01:55:37.807879 11779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.190:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.125.190
--webserver_port=45449
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:37.809314 11779 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:37.810883 11779 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:37.818380 11784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:37.818478 11785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:37.818734 11787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:37.819242 11779 server_base.cc:1047] running on GCE node
I20251025 01:55:37.819957 11779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:37.821341 11779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:37.822557 11779 hybrid_clock.cc:648] HybridClock initialized: now 1761357337822514 us; error 36 us; skew 500 ppm
I20251025 01:55:37.822878 11779 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:37.825044 11779 webserver.cc:492] Webserver started at http://127.9.125.190:45449/ using document root <none> and password file <none>
I20251025 01:55:37.825767 11779 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:37.825868 11779 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:37.830919 11779 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20251025 01:55:37.833477 11793 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:37.834641 11779 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251025 01:55:37.834781 11779 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
uuid: "76559300dae44c0d8a17b583a8875b3f"
format_stamp: "Formatted at 2025-10-25 01:55:14 on dist-test-slave-mkd1"
I20251025 01:55:37.835460 11779 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:37.846794 11779 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:37.847517 11779 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:37.847764 11779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:37.867733 11779 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.190:35925
I20251025 01:55:37.867753 11847 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.190:35925 every 8 connection(s)
I20251025 01:55:37.869313 11779 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-0/data/info.pb
I20251025 01:55:37.871331 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 11779
I20251025 01:55:37.876602 11848 sys_catalog.cc:263] Verifying existing consensus state
I20251025 01:55:37.879891 11848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Bootstrap starting.
I20251025 01:55:37.896055 11848 log.cc:826] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:37.966621 11848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=102 overwritten=0 applied=102 ignored=0} inserts{seen=75 ignored=0} mutations{seen=92 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251025 01:55:37.967170 11848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Bootstrap complete.
I20251025 01:55:37.972272 11848 raft_consensus.cc:359] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:37.972920 11848 raft_consensus.cc:740] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76559300dae44c0d8a17b583a8875b3f, State: Initialized, Role: FOLLOWER
I20251025 01:55:37.973516 11848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 102, Last appended: 8.102, Last appended by leader: 102, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:37.974308 11852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:37.974608 11852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:37.975304 11848 sys_catalog.cc:565] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: configured and running, proceeding with master startup.
I20251025 01:55:37.990690 11866 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:37.990772 11866 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: loading cluster ID for follower catalog manager: success
I20251025 01:55:37.997440 11866 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: acquiring CA information for follower catalog manager: success
I20251025 01:55:37.995278 11779 master_runner.cc:186] Error getting master registration for 127.9.125.189:45553: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111),
I20251025 01:55:38.001159 11779 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:38.009526 11866 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 01:55:38.015069 10370 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:38.016822 10508 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:38.018968 11809 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" instance_seqno: 1761357316719619) as {username='slave'} at 127.9.125.129:38305; Asking this server to re-register.
I20251025 01:55:38.019011 11810 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" instance_seqno: 1761357317153916) as {username='slave'} at 127.9.125.130:43355; Asking this server to re-register.
I20251025 01:55:38.019780 10508 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:38.019917 10370 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:38.021780 11809 ts_manager.cc:194] Registered new tserver with Master: 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:38.021896 11810 ts_manager.cc:194] Registered new tserver with Master: c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:38.369632 10644 heartbeater.cc:344] Connected to a master server at 127.9.125.190:35925
I20251025 01:55:38.371097 11810 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" instance_seqno: 1761357317567829) as {username='slave'} at 127.9.125.131:49131; Asking this server to re-register.
I20251025 01:55:38.371678 10644 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:38.372838 11810 ts_manager.cc:194] Registered new tserver with Master: 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:38.430665 11852 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:38.430925 11852 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:38.432749 11852 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:38.433250 11796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:55:38.435209 11796 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:55:38.435555 11794 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:38.435678 11794 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:38.435994 11852 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251025 01:55:38.968837 11881 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:38.969017 11881 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:38.969684 11881 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:38.971993 11796 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:55:38.972218 11794 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:38.972345 11794 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:38.972642 11881 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251025 01:55:39.897451 11890 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:39.897620 11890 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:39.898208 11890 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:39.900148 11796 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:55:39.900338 11794 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:39.900444 11794 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:39.900622 11890 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251025 01:55:40.405905 11899 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:40.406072 11899 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:40.406611 11899 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:40.408385 11796 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:55:40.408589 11794 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:40.408697 11794 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76559300dae44c0d8a17b583a8875b3f; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:40.408928 11899 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251025 01:55:41.057468 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.189:45553
--webserver_interface=127.9.125.189
--webserver_port=40341
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251025 01:55:41.376129 11913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:41.376442 11913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:41.376519 11913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 01:55:41.384264 11913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 01:55:41.384344 11913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:41.384405 11913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 01:55:41.384466 11913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 01:55:41.394026 11913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.189:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.125.189
--webserver_port=40341
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:41.395581 11913 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:41.397238 11913 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:41.405153 11921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:41.405088 11920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:41.405886 11923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:41.406148 11913 server_base.cc:1047] running on GCE node
I20251025 01:55:41.406793 11913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:41.408080 11913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:41.409292 11913 hybrid_clock.cc:648] HybridClock initialized: now 1761357341409246 us; error 34 us; skew 500 ppm
I20251025 01:55:41.409639 11913 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:41.412245 11913 webserver.cc:492] Webserver started at http://127.9.125.189:40341/ using document root <none> and password file <none>
I20251025 01:55:41.412961 11913 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:41.413106 11913 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:41.418421 11913 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20251025 01:55:41.421119 11929 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:41.422379 11913 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251025 01:55:41.422533 11913 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce"
format_stamp: "Formatted at 2025-10-25 01:55:15 on dist-test-slave-mkd1"
I20251025 01:55:41.423378 11913 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:41.441084 11913 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:41.441979 11913 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:41.442247 11913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:41.459847 11913 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.189:45553
I20251025 01:55:41.459877 11980 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.189:45553 every 8 connection(s)
I20251025 01:55:41.461426 11913 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data/info.pb
I20251025 01:55:41.465436 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 11913
I20251025 01:55:41.468881 11981 sys_catalog.cc:263] Verifying existing consensus state
I20251025 01:55:41.473546 11981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Bootstrap starting.
I20251025 01:55:41.490362 11981 log.cc:826] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:41.502585 10509 heartbeater.cc:344] Connected to a master server at 127.9.125.189:45553
I20251025 01:55:41.519999 10371 heartbeater.cc:344] Connected to a master server at 127.9.125.189:45553
I20251025 01:55:41.559489 11981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=91 overwritten=0 applied=88 ignored=0} inserts{seen=67 ignored=0} mutations{seen=72 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20251025 01:55:41.560035 11981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Bootstrap complete.
I20251025 01:55:41.565315 11981 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 5 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:41.568116 11981 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, State: Initialized, Role: FOLLOWER
I20251025 01:55:41.568704 11981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 88, Last appended: 5.91, Last appended by leader: 91, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:41.569437 11990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:41.569679 11990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:41.570385 11981 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: configured and running, proceeding with master startup.
I20251025 01:55:41.594797 12004 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:41.594893 12004 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: loading cluster ID for follower catalog manager: success
I20251025 01:55:41.599140 12004 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: acquiring CA information for follower catalog manager: success
I20251025 01:55:41.601656 12004 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 01:55:41.620963 11913 master_runner.cc:186] Error getting master registration for 127.9.125.188:36641: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111),
I20251025 01:55:41.624719 11913 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:41.883122 10645 heartbeater.cc:344] Connected to a master server at 127.9.125.189:45553
I20251025 01:55:41.885495 11946 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" instance_seqno: 1761357317567829) as {username='slave'} at 127.9.125.131:45407; Asking this server to re-register.
I20251025 01:55:41.886118 10645 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:41.887813 11946 ts_manager.cc:194] Registered new tserver with Master: 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:42.007361 11946 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" instance_seqno: 1761357317153916) as {username='slave'} at 127.9.125.130:49695; Asking this server to re-register.
I20251025 01:55:42.007973 10509 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:42.009140 11946 ts_manager.cc:194] Registered new tserver with Master: 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:42.024819 11946 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" instance_seqno: 1761357316719619) as {username='slave'} at 127.9.125.129:36331; Asking this server to re-register.
I20251025 01:55:42.025419 10371 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:42.026552 11946 ts_manager.cc:194] Registered new tserver with Master: c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:42.037751 11990 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:42.037947 11990 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:42.039980 11990 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:42.040483 11820 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 91 } } ignore_live_leader: false dest_uuid: "76559300dae44c0d8a17b583a8875b3f" is_pre_election: true
I20251025 01:55:42.040958 11820 raft_consensus.cc:2368] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ceaf0e1ce0f4c23b39a5bf4736de5ce for earlier term 6. Current term is 8.
W20251025 01:55:42.040907 11930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:42.041759 11932 leader_election.cc:400] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 6 pre-election: Vote denied by peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ceaf0e1ce0f4c23b39a5bf4736de5ce for earlier term 6. Current term is 8.
I20251025 01:55:42.041893 11932 leader_election.cc:403] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20251025 01:55:42.042222 11990 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 5 FOLLOWER]: Advancing to term 8
W20251025 01:55:42.042618 11930 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:42.045598 11990 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 8 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ceaf0e1ce0f4c23b39a5bf4736de5ce for earlier term 6. Current term is 8.
I20251025 01:55:42.645258 12016 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:42.645519 12016 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:42.646199 12016 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:42.646771 11820 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 9 candidate_status { last_received { term: 5 index: 91 } } ignore_live_leader: false dest_uuid: "76559300dae44c0d8a17b583a8875b3f" is_pre_election: true
I20251025 01:55:42.647128 11820 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ceaf0e1ce0f4c23b39a5bf4736de5ce for term 9 because replica has last-logged OpId of term: 8 index: 102, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 91.
W20251025 01:55:42.648205 11930 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:42.648376 11930 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce; no voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:42.648664 12016 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251025 01:55:42.722621 12019 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:42.722806 12019 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:42.723503 12019 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:42.725987 11794 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:42.730163 11956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 102 } } ignore_live_leader: false dest_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" is_pre_election: true
I20251025 01:55:42.730620 11956 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76559300dae44c0d8a17b583a8875b3f in term 8.
I20251025 01:55:42.731309 11796 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f; no voters: 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:42.731623 12019 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251025 01:55:42.731747 12019 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:42.731848 12019 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 8 FOLLOWER]: Advancing to term 9
I20251025 01:55:42.735224 12019 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:42.735834 12019 leader_election.cc:290] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 election: Requested vote from peers 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:42.736274 11956 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76559300dae44c0d8a17b583a8875b3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 102 } } ignore_live_leader: false dest_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce"
I20251025 01:55:42.736444 11956 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 8 FOLLOWER]: Advancing to term 9
W20251025 01:55:42.738054 11794 leader_election.cc:336] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:42.739409 11956 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76559300dae44c0d8a17b583a8875b3f in term 9.
I20251025 01:55:42.739801 11796 leader_election.cc:304] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f; no voters: 286b4c3025e24a7c8107763fa40d3ab1
I20251025 01:55:42.740074 12019 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 9 FOLLOWER]: Leader election won for term 9
I20251025 01:55:42.740573 12019 raft_consensus.cc:697] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [term 9 LEADER]: Becoming Leader. State: Replica: 76559300dae44c0d8a17b583a8875b3f, State: Running, Role: LEADER
I20251025 01:55:42.741031 12019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 8.102, Last appended by leader: 102, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:42.743706 12024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76559300dae44c0d8a17b583a8875b3f. Latest consensus state: current_term: 9 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:42.743980 12024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:42.744606 12026 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 01:55:42.747340 12026 catalog_manager.cc:679] Loaded metadata for table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a]
I20251025 01:55:42.747752 12026 catalog_manager.cc:679] Loaded metadata for table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b]
I20251025 01:55:42.748086 12026 catalog_manager.cc:679] Loaded metadata for table default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc]
I20251025 01:55:42.748451 12026 catalog_manager.cc:679] Loaded metadata for table default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889]
I20251025 01:55:42.748847 12026 catalog_manager.cc:679] Loaded metadata for table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c]
I20251025 01:55:42.749292 12026 catalog_manager.cc:679] Loaded metadata for table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]
I20251025 01:55:42.749756 12026 catalog_manager.cc:679] Loaded metadata for table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980]
I20251025 01:55:42.750285 12026 catalog_manager.cc:679] Loaded metadata for table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]
I20251025 01:55:42.750576 12026 catalog_manager.cc:679] Loaded metadata for table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f]
I20251025 01:55:42.750847 12026 catalog_manager.cc:679] Loaded metadata for table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]
I20251025 01:55:42.751130 12026 catalog_manager.cc:679] Loaded metadata for table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]
I20251025 01:55:42.751600 12026 catalog_manager.cc:679] Loaded metadata for table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811]
I20251025 01:55:42.751861 12026 catalog_manager.cc:679] Loaded metadata for table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652]
I20251025 01:55:42.752126 12026 catalog_manager.cc:679] Loaded metadata for table default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737]
I20251025 01:55:42.752385 12026 catalog_manager.cc:679] Loaded metadata for table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2]
I20251025 01:55:42.752637 12026 catalog_manager.cc:679] Loaded metadata for table default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0]
I20251025 01:55:42.752898 12026 catalog_manager.cc:679] Loaded metadata for table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a]
I20251025 01:55:42.755146 12026 tablet_loader.cc:96] loaded metadata for tablet 0c92525ba189465fa7eff84f166da3d0 (table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811])
I20251025 01:55:42.755470 12026 tablet_loader.cc:96] loaded metadata for tablet 0f9f9151de784d6d9e6ad0d75e76fede (table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b])
I20251025 01:55:42.755856 12026 tablet_loader.cc:96] loaded metadata for tablet 174f2bfde6ef4ab4b90750c0f63fa9a1 (table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980])
I20251025 01:55:42.756242 12026 tablet_loader.cc:96] loaded metadata for tablet 30ab28ea6ce847778e877ca0915448a6 (table default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889])
I20251025 01:55:42.756600 12026 tablet_loader.cc:96] loaded metadata for tablet 3a1411d1fa224fb9be8345d6daca1bcd (table default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc])
I20251025 01:55:42.756839 12026 tablet_loader.cc:96] loaded metadata for tablet 3d37511aec3746cea09c1ba224cb14db (table default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0])
I20251025 01:55:42.757215 12026 tablet_loader.cc:96] loaded metadata for tablet 4e736e021c9d4855acbe0f56c1aa756f (table default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737])
I20251025 01:55:42.757853 12026 tablet_loader.cc:96] loaded metadata for tablet 87aaf87a6c284ca980e5fb6f896c5e4e (table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a])
I20251025 01:55:42.758646 12026 tablet_loader.cc:96] loaded metadata for tablet 937b8bb578ac4fdba8c1e800df5d6daa (table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a])
I20251025 01:55:42.759821 12026 tablet_loader.cc:96] loaded metadata for tablet a600dcbd31104aee98146b72ee257034 (table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c])
I20251025 01:55:42.760277 12026 tablet_loader.cc:96] loaded metadata for tablet b3578b8a1d8f412d9a8b2b968bb9c583 (table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2])
I20251025 01:55:42.760864 12026 tablet_loader.cc:96] loaded metadata for tablet cceaea14d4054de09cdaacb93f9b5582 (table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f])
I20251025 01:55:42.761300 12026 tablet_loader.cc:96] loaded metadata for tablet ce8decc3a0fc4542bd94062e19c8fa55 (table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812])
I20251025 01:55:42.761545 12026 tablet_loader.cc:96] loaded metadata for tablet d7df9d477cf1448d91cd8cac0ab321a9 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5])
I20251025 01:55:42.762339 12026 tablet_loader.cc:96] loaded metadata for tablet eab40c1346c54c719060421fc54f8be4 (table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652])
I20251025 01:55:42.762593 12026 tablet_loader.cc:96] loaded metadata for tablet effc501d8ef24438a3fd7c30feb32f87 (table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433])
I20251025 01:55:42.763015 12026 tablet_loader.cc:96] loaded metadata for tablet fc9e67e3872b4990b0d0181eb96ad98c (table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b])
I20251025 01:55:42.763334 12026 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 01:55:42.764721 12026 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:42.764793 12026 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 01:55:42.766680 12026 catalog_manager.cc:1514] Loading token signing keys...
I20251025 01:55:42.768097 12026 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f: Loaded TSK: 0
I20251025 01:55:42.768859 12026 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 01:55:42.770215 12026 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251025 01:55:42.771721 12026 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: 23
I20251025 01:55:42.828127 11810 master_service.cc:954] ReplaceTablet: received request to replace tablet 33df938c44154f41b333523a51b65e4e from {username='slave'} at 127.0.0.1:37534
I20251025 01:55:42.828173 11809 master_service.cc:954] ReplaceTablet: received request to replace tablet f13a1ad12e2f4f2c95103cf8d76d6da6 from {username='slave'} at 127.0.0.1:37534
I20251025 01:55:42.828953 11808 master_service.cc:954] ReplaceTablet: received request to replace tablet 3d37511aec3746cea09c1ba224cb14db from {username='slave'} at 127.0.0.1:37534
I20251025 01:55:42.829047 11806 master_service.cc:954] ReplaceTablet: received request to replace tablet cceaea14d4054de09cdaacb93f9b5582 from {username='slave'} at 127.0.0.1:37534
I20251025 01:55:42.829447 11807 master_service.cc:954] ReplaceTablet: received request to replace tablet 5a2f9794447f4d7e8f6f548e4b26c8c6 from {username='slave'} at 127.0.0.1:37534
I20251025 01:55:42.829373 11803 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37534:
table { table_name: "default.Table_8553cdea51804bfa861b6c10537b99c3" } new_table_name: "default.Table_89a70ae51ea04c43b2f1842ec3f87253" modify_external_catalogs: true
I20251025 01:55:42.829520 11804 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37534:
table { table_name: "default.Table_9187de27a70f48e08cc7284131fe543d" } modify_external_catalogs: true
I20251025 01:55:42.829744 11804 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37534:
table { table_name: "default.Table_9187de27a70f48e08cc7284131fe543d" } modify_external_catalogs: true
I20251025 01:55:42.830390 11805 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37534:
name: "default.Table_50c0606a959945b3a1edd9df467a65dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:42.831482 11802 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37534:
name: "default.Table_ac8050cb49d54bc2a444d81cf8492c75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:42.832156 11807 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37534:
name: "default.Table_20c488ac6bf24db48ea2ab061d253e7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:42.832691 11810 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37534:
name: "default.Table_a981c91dba574e1c8ccce55a41c4bd88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:42.833305 11801 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37534:
name: "default.Table_9328a3e2d017427796e74bbabc9f07bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:42.833510 11809 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37534:
table { table_name: "default.Table_ac99b534589f45e6871ddfa3a1331b16" } new_table_name: "default.Table_5180205af52d470586f94f073f3537ca" modify_external_catalogs: true
F20251025 01:55:42.836521 11865 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 4 out of 13 received events: NotificationEvent(eventId=24, eventTime=1761357321, eventType=CREATE_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_a981c91dba574e1c8ccce55a41c4bd88, message={"server":"","servicePrincipal":"","db":"default","table":"table_a981c91dba574e1c8ccce55a41c4bd88","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a981c91dba574e1c8ccce55a41c4bd88\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357320},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_a981c91dba574e1c8ccce55a41c4bd88\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a981c91dba574e1c8ccce55a41c4bd88\",\"kudu.table_id\":\"269cfd64fbd94f0486843525917d9bb4\",\"transient_lastDdlTime\":\"1761357320\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_44e4ccc4f2de47198a0599e2577108cf, message={"server":"","servicePrincipal":"","db":"default","table":"table_44e4ccc4f2de47198a0599e2577108cf","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_44e4ccc4f2de47198a0599e2577108cf\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357319},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_44e4ccc4f2de47198a0599e2577108cf\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_44e4ccc4f2de47198a0599e2577108cf\",\"kudu.table_id\":\"58afab4104e84c868eeccda411b8452b\",\"transient_lastDdlTime\":\"1761357319\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251025 01:55:42.838083 11956 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Refusing update from remote peer 76559300dae44c0d8a17b583a8875b3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 91. Preceding OpId from leader: term: 9 index: 104. (index mismatch)
I20251025 01:55:42.838972 12024 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 88, Time since last communication: 0.000s
@ 0x7f9241a80dcd google::LogMessage::Fail() at ??:0
W20251025 01:55:42.839557 11794 consensus_peers.cc:597] T 00000000000000000000000000000000 P 76559300dae44c0d8a17b583a8875b3f -> Peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Couldn't send request to peer 286b4c3025e24a7c8107763fa40d3ab1. Status: Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f9241a84b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f9241a807cc google::LogMessage::Flush() at ??:0
@ 0x7f9241a80a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9258439b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f9258438411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f925844008c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f925843ff09 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f92447f4ee9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9247dfc6db start_thread at ??:0
I20251025 01:55:42.891117 12034 mvcc.cc:204] Tried to move back new op lower bound from 7214519644362174464 to 7214519613173293056. Current Snapshot: MvccSnapshot[applied={T|T < 7214519644362174464 or (T in {7214519644362174464})}]
@ 0x7f924093171f clone at ??:0
I20251025 01:55:42.894362 12016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76559300dae44c0d8a17b583a8875b3f. Latest consensus state: current_term: 9 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:42.894647 12016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:42.894433 12035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:42.894779 12035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:42.895465 12038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:42.895764 12038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:42.896008 12036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "76559300dae44c0d8a17b583a8875b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:42.896252 12036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: FOLLOWER
W20251025 01:55:43.050911 11931 connection.cc:537] server connection from 127.0.0.1:41708 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:43.051375 10397 connection.cc:537] client connection to 127.9.125.190:35925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:43.051388 10651 connection.cc:537] client connection to 127.9.125.190:35925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:43.051584 10256 connection.cc:537] client connection to 127.9.125.190:35925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:43.051972 10651 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.125.190:35925)
W20251025 01:55:43.052395 10651 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.125.190:35925)
W20251025 01:55:43.052577 10651 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.125.190:35925)
W20251025 01:55:43.052963 10651 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.125.190:35925)
I20251025 01:55:43.343993 12036 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 76559300dae44c0d8a17b583a8875b3f)
I20251025 01:55:43.344225 12036 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:43.344919 12036 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:43.346930 11932 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:43.347097 11930 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:43.347218 11930 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce; no voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:43.347509 12036 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251025 01:55:43.823354 9718 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu pid 11913
I20251025 01:55:43.829571 11976 generic_service.cc:196] Checking for leaks (request via RPC)
I20251025 01:55:43.933960 9718 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu with pid 11913
I20251025 01:55:44.524678 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.189:45553
--webserver_interface=127.9.125.189
--webserver_port=40341
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251025 01:55:44.837982 12061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:44.838266 12061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:44.838395 12061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 01:55:44.846146 12061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 01:55:44.846237 12061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:44.846324 12061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 01:55:44.846386 12061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 01:55:44.855809 12061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.189:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.125.189
--webserver_port=40341
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:44.857474 12061 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:44.859185 12061 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:44.866863 12068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:44.866839 12067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:44.867552 12070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:44.868773 12061 server_base.cc:1047] running on GCE node
I20251025 01:55:44.869486 12061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:44.870883 12061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:44.872100 12061 hybrid_clock.cc:648] HybridClock initialized: now 1761357344872061 us; error 33 us; skew 500 ppm
I20251025 01:55:44.872418 12061 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:44.874943 12061 webserver.cc:492] Webserver started at http://127.9.125.189:40341/ using document root <none> and password file <none>
I20251025 01:55:44.875681 12061 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:44.875782 12061 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:44.880636 12061 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20251025 01:55:44.883339 12076 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 01:55:44.884582 12061 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20251025 01:55:44.884735 12061 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce"
format_stamp: "Formatted at 2025-10-25 01:55:15 on dist-test-slave-mkd1"
I20251025 01:55:44.885360 12061 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:44.916005 12061 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:44.916806 12061 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:44.917047 12061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:44.935653 12061 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.189:45553
I20251025 01:55:44.935704 12127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.189:45553 every 8 connection(s)
I20251025 01:55:44.937337 12061 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-1/data/info.pb
I20251025 01:55:44.942723 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 12061
I20251025 01:55:44.946884 12128 sys_catalog.cc:263] Verifying existing consensus state
I20251025 01:55:44.952122 12128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Bootstrap starting.
I20251025 01:55:44.952935 10645 heartbeater.cc:344] Connected to a master server at 127.9.125.189:45553
I20251025 01:55:44.968538 12128 log.cc:826] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:45.024727 10509 heartbeater.cc:344] Connected to a master server at 127.9.125.189:45553
I20251025 01:55:45.041962 12128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=105 overwritten=0 applied=102 ignored=0} inserts{seen=75 ignored=0} mutations{seen=92 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20251025 01:55:45.042748 10371 heartbeater.cc:344] Connected to a master server at 127.9.125.189:45553
I20251025 01:55:45.042810 12128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Bootstrap complete.
I20251025 01:55:45.048249 12128 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:45.050974 12128 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, State: Initialized, Role: FOLLOWER
I20251025 01:55:45.051767 12128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 102, Last appended: 9.105, Last appended by leader: 105, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:45.052456 12134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:45.052706 12134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:45.053406 12128 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: configured and running, proceeding with master startup.
I20251025 01:55:45.070093 12061 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:55:45.076604 12061 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:45.079613 12149 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:45.079761 12149 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: loading cluster ID for follower catalog manager: success
I20251025 01:55:45.085321 12149 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: acquiring CA information for follower catalog manager: success
I20251025 01:55:45.089265 12149 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 01:55:45.457304 12093 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" instance_seqno: 1761357317567829) as {username='slave'} at 127.9.125.131:45517; Asking this server to re-register.
I20251025 01:55:45.458001 10645 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:45.459750 12093 ts_manager.cc:194] Registered new tserver with Master: 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:45.528105 12093 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" instance_seqno: 1761357317153916) as {username='slave'} at 127.9.125.130:48335; Asking this server to re-register.
I20251025 01:55:45.528668 10509 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:45.529848 12093 ts_manager.cc:194] Registered new tserver with Master: 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:45.545614 12093 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" instance_seqno: 1761357316719619) as {username='slave'} at 127.9.125.129:47821; Asking this server to re-register.
I20251025 01:55:45.546130 10371 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:45.547302 12093 ts_manager.cc:194] Registered new tserver with Master: c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:45.672943 12152 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:45.673157 12152 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:45.675153 12152 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:45.676159 12079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:45.677248 12079 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:45.677451 12077 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:45.677597 12077 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce; no voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:45.677928 12152 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251025 01:55:46.407819 12159 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:46.407977 12159 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:46.408664 12159 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:46.412222 12079 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:46.412369 12077 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:46.412509 12077 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce; no voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:46.412740 12159 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251025 01:55:46.975634 12163 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:46.975816 12163 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:46.976519 12163 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:46.978593 12079 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:46.978765 12077 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:46.978886 12077 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce; no voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:46.979138 12163 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251025 01:55:48.171648 9718 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
/tmp/dist-test-tasknR9X9f/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.188:36641
--webserver_interface=127.9.125.188
--webserver_port=33371
--builtin_ntp_servers=127.9.125.148:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251025 01:55:48.477845 12172 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:48.478039 12172 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:48.478709 12172 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:48.481060 12079 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:48.481241 12077 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641): Network error: Client connection negotiation failed: client connection to 127.9.125.188:36641: connect: Connection refused (error 111)
I20251025 01:55:48.481324 12077 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce; no voters: 286b4c3025e24a7c8107763fa40d3ab1, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:48.481556 12172 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251025 01:55:48.493155 12170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 01:55:48.493439 12170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 01:55:48.493521 12170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 01:55:48.501888 12170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 01:55:48.501994 12170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 01:55:48.502068 12170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 01:55:48.502135 12170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 01:55:48.511514 12170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.125.148:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:38295
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.125.188:36641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.125.188
--webserver_port=33371
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 01:43:21 UTC on 24a791456cd2
build id 8704
ASAN enabled
I20251025 01:55:48.513142 12170 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 01:55:48.514787 12170 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 01:55:48.522084 12180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:48.522112 12181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 01:55:48.522770 12183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 01:55:48.523881 12170 server_base.cc:1047] running on GCE node
I20251025 01:55:48.524549 12170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 01:55:48.525807 12170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 01:55:48.526952 12170 hybrid_clock.cc:648] HybridClock initialized: now 1761357348526914 us; error 30 us; skew 500 ppm
I20251025 01:55:48.527310 12170 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 01:55:48.529424 12170 webserver.cc:492] Webserver started at http://127.9.125.188:33371/ using document root <none> and password file <none>
I20251025 01:55:48.530202 12170 fs_manager.cc:362] Metadata directory not provided
I20251025 01:55:48.530293 12170 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 01:55:48.535317 12170 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20251025 01:55:48.543898 12189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.001s user 0.000s sys 0.000s
I20251025 01:55:48.545342 12170 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
I20251025 01:55:48.545473 12170 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data,/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
uuid: "286b4c3025e24a7c8107763fa40d3ab1"
format_stamp: "Formatted at 2025-10-25 01:55:15 on dist-test-slave-mkd1"
I20251025 01:55:48.546095 12170 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 01:55:48.573140 12170 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 01:55:48.573863 12170 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 01:55:48.574083 12170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 01:55:48.595861 12170 rpc_server.cc:307] RPC server started. Bound to: 127.9.125.188:36641
I20251025 01:55:48.595882 12240 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.125.188:36641 every 8 connection(s)
I20251025 01:55:48.597723 12170 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/master-2/data/info.pb
I20251025 01:55:48.602607 9718 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknR9X9f/build/asan/bin/kudu as pid 12170
I20251025 01:55:48.604847 12241 sys_catalog.cc:263] Verifying existing consensus state
I20251025 01:55:48.608881 12241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Bootstrap starting.
I20251025 01:55:48.626659 12241 log.cc:826] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Log is configured to *not* fsync() on all Append() calls
I20251025 01:55:48.692854 12241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Bootstrap replayed 1/1 log segments. Stats: ops{read=103 overwritten=1 applied=102 ignored=0} inserts{seen=75 ignored=0} mutations{seen=92 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251025 01:55:48.693370 12241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: Bootstrap complete.
I20251025 01:55:48.699219 12241 raft_consensus.cc:359] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:48.699843 12241 raft_consensus.cc:740] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286b4c3025e24a7c8107763fa40d3ab1, State: Initialized, Role: FOLLOWER
I20251025 01:55:48.700405 12241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 102, Last appended: 8.102, Last appended by leader: 102, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:48.701393 12245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:48.701766 12245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:48.702577 12241 sys_catalog.cc:565] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 01:55:48.712220 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:55:48.717121 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:48.719743 12259 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:48.719894 12259 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: loading cluster ID for follower catalog manager: success
I20251025 01:55:48.725473 12259 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: acquiring CA information for follower catalog manager: success
I20251025 01:55:48.729019 12259 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 01:55:48.910638 10643 heartbeater.cc:344] Connected to a master server at 127.9.125.188:36641
I20251025 01:55:48.914728 12206 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" instance_seqno: 1761357317567829) as {username='slave'} at 127.9.125.131:44565; Asking this server to re-register.
I20251025 01:55:48.915570 10643 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:48.917533 12206 ts_manager.cc:194] Registered new tserver with Master: 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021)
I20251025 01:55:49.018935 10507 heartbeater.cc:344] Connected to a master server at 127.9.125.188:36641
I20251025 01:55:49.020841 12206 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" instance_seqno: 1761357317153916) as {username='slave'} at 127.9.125.130:40557; Asking this server to re-register.
I20251025 01:55:49.021440 10507 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:49.022927 12206 ts_manager.cc:194] Registered new tserver with Master: 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
I20251025 01:55:49.036451 10369 heartbeater.cc:344] Connected to a master server at 127.9.125.188:36641
I20251025 01:55:49.038103 12206 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" instance_seqno: 1761357316719619) as {username='slave'} at 127.9.125.129:50451; Asking this server to re-register.
I20251025 01:55:49.038693 10369 heartbeater.cc:461] Registering TS with master...
I20251025 01:55:49.040055 12206 ts_manager.cc:194] Registered new tserver with Master: c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581)
I20251025 01:55:49.137475 12245 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:49.137702 12245 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:49.139451 12245 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:49.140022 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:49.142356 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
I20251025 01:55:49.146122 12103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "286b4c3025e24a7c8107763fa40d3ab1" candidate_term: 9 candidate_status { last_received { term: 8 index: 102 } } ignore_live_leader: false dest_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" is_pre_election: true
I20251025 01:55:49.146510 12103 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 286b4c3025e24a7c8107763fa40d3ab1 in current term 9: Already voted for candidate 76559300dae44c0d8a17b583a8875b3f in this term.
I20251025 01:55:49.147254 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:49.147621 12245 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 8 FOLLOWER]: Advancing to term 9
I20251025 01:55:49.151496 12245 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251025 01:55:49.803426 12270 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:49.803639 12270 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:49.804473 12270 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
I20251025 01:55:49.805323 12103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "286b4c3025e24a7c8107763fa40d3ab1" candidate_term: 10 candidate_status { last_received { term: 8 index: 102 } } ignore_live_leader: false dest_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" is_pre_election: true
I20251025 01:55:49.805611 12103 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 286b4c3025e24a7c8107763fa40d3ab1 for term 10 because replica has last-logged OpId of term: 9 index: 105, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 102.
W20251025 01:55:49.806110 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
I20251025 01:55:49.806403 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:49.806645 12270 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251025 01:55:50.078255 12061 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:55:50.078457 12061 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:50.617257 12280 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:50.617465 12280 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:50.618076 12280 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
I20251025 01:55:50.619277 12103 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "286b4c3025e24a7c8107763fa40d3ab1" candidate_term: 10 candidate_status { last_received { term: 8 index: 102 } } ignore_live_leader: false dest_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" is_pre_election: true
I20251025 01:55:50.619581 12103 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 286b4c3025e24a7c8107763fa40d3ab1 for term 10 because replica has last-logged OpId of term: 9 index: 105, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 102.
W20251025 01:55:50.620043 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
I20251025 01:55:50.620178 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:50.620432 12280 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251025 01:55:50.637418 12283 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:50.637579 12283 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:50.638159 12283 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
W20251025 01:55:50.640170 12079 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
I20251025 01:55:50.644575 12216 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 10 candidate_status { last_received { term: 9 index: 105 } } ignore_live_leader: false dest_uuid: "286b4c3025e24a7c8107763fa40d3ab1" is_pre_election: true
I20251025 01:55:50.644975 12216 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ceaf0e1ce0f4c23b39a5bf4736de5ce in term 9.
I20251025 01:55:50.645663 12077 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1; no voters: 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:50.645987 12283 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251025 01:55:50.646111 12283 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 01:55:50.646201 12283 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 9 FOLLOWER]: Advancing to term 10
I20251025 01:55:50.649643 12283 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:50.650252 12283 leader_election.cc:290] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 election: Requested vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 286b4c3025e24a7c8107763fa40d3ab1 (127.9.125.188:36641)
I20251025 01:55:50.650818 12216 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" candidate_term: 10 candidate_status { last_received { term: 9 index: 105 } } ignore_live_leader: false dest_uuid: "286b4c3025e24a7c8107763fa40d3ab1"
I20251025 01:55:50.650990 12216 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 9 FOLLOWER]: Advancing to term 10
W20251025 01:55:50.651575 12079 leader_election.cc:336] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
I20251025 01:55:50.653138 12216 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ceaf0e1ce0f4c23b39a5bf4736de5ce in term 10.
I20251025 01:55:50.653438 12077 leader_election.cc:304] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 286b4c3025e24a7c8107763fa40d3ab1; no voters: 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:50.653654 12283 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 10 FOLLOWER]: Leader election won for term 10
I20251025 01:55:50.654007 12283 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [term 10 LEADER]: Becoming Leader. State: Replica: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, State: Running, Role: LEADER
I20251025 01:55:50.654489 12283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 9.105, Last appended by leader: 105, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:50.656569 12288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce. Latest consensus state: current_term: 10 leader_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:50.656878 12288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:50.877472 9718 master-stress-test.cc:577] Killing test thread 1/25
W20251025 01:55:51.050453 12079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20251025 01:55:51.051525 12079 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce -> Peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Couldn't send request to peer 76559300dae44c0d8a17b583a8875b3f. Status: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 01:55:51.216547 12216 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Refusing update from remote peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Log matching property violated. Preceding OpId in replica: term: 8 index: 102. Preceding OpId from leader: term: 10 index: 106. (index mismatch)
I20251025 01:55:51.217283 12288 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 102, Time since last communication: 0.000s
I20251025 01:55:51.223281 12295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce. Latest consensus state: current_term: 10 leader_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:51.223524 12295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:51.227999 12288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:51.228235 12288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:51.228123 12304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:51.228268 12295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:51.228402 12304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:51.228410 12295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 01:55:51.228678 12298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } } }
I20251025 01:55:51.228842 12298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce [sys.catalog]: This master's current role is: LEADER
I20251025 01:55:51.301921 12290 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 01:55:51.305222 12290 catalog_manager.cc:679] Loaded metadata for table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a]
I20251025 01:55:51.305688 12290 catalog_manager.cc:679] Loaded metadata for table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b]
I20251025 01:55:51.306072 12290 catalog_manager.cc:679] Loaded metadata for table default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc]
I20251025 01:55:51.306447 12290 catalog_manager.cc:679] Loaded metadata for table default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889]
I20251025 01:55:51.306777 12290 catalog_manager.cc:679] Loaded metadata for table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c]
I20251025 01:55:51.307106 12290 catalog_manager.cc:679] Loaded metadata for table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812]
I20251025 01:55:51.307448 12290 catalog_manager.cc:679] Loaded metadata for table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980]
I20251025 01:55:51.307888 12290 catalog_manager.cc:679] Loaded metadata for table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433]
I20251025 01:55:51.308194 12290 catalog_manager.cc:679] Loaded metadata for table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f]
I20251025 01:55:51.308497 12290 catalog_manager.cc:679] Loaded metadata for table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b]
I20251025 01:55:51.308792 12290 catalog_manager.cc:679] Loaded metadata for table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5]
I20251025 01:55:51.309240 12290 catalog_manager.cc:679] Loaded metadata for table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811]
I20251025 01:55:51.309543 12290 catalog_manager.cc:679] Loaded metadata for table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652]
I20251025 01:55:51.309844 12290 catalog_manager.cc:679] Loaded metadata for table default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737]
I20251025 01:55:51.310133 12290 catalog_manager.cc:679] Loaded metadata for table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2]
I20251025 01:55:51.310429 12290 catalog_manager.cc:679] Loaded metadata for table default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0]
I20251025 01:55:51.310708 12290 catalog_manager.cc:679] Loaded metadata for table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a]
I20251025 01:55:51.313118 12290 tablet_loader.cc:96] loaded metadata for tablet 0c92525ba189465fa7eff84f166da3d0 (table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811])
I20251025 01:55:51.313472 12290 tablet_loader.cc:96] loaded metadata for tablet 0f9f9151de784d6d9e6ad0d75e76fede (table default.table_986ff2b14daa4beebda43706d47dc98b [id=10c6d32941e545699b3285ec0b50f78b])
I20251025 01:55:51.313916 12290 tablet_loader.cc:96] loaded metadata for tablet 174f2bfde6ef4ab4b90750c0f63fa9a1 (table default.table_dfca34fd2dec49089ebdfe61dcc20ba8 [id=46a03684abfd437997c4881af2f5b980])
I20251025 01:55:51.314352 12290 tablet_loader.cc:96] loaded metadata for tablet 30ab28ea6ce847778e877ca0915448a6 (table default.table_6b948e527acb484d8c82d7ea153e9304 [id=35ac4045482e4025b49f122add8f0889])
I20251025 01:55:51.314769 12290 tablet_loader.cc:96] loaded metadata for tablet 3a1411d1fa224fb9be8345d6daca1bcd (table default.table_9187de27a70f48e08cc7284131fe543d [id=25e9ab81dd8247c2a0f8bf9913e237bc])
I20251025 01:55:51.315410 12290 tablet_loader.cc:96] loaded metadata for tablet 4e736e021c9d4855acbe0f56c1aa756f (table default.table_547c7e192439428aa78c62d503e17536 [id=c799aad2d817457cb9d21865a082a737])
I20251025 01:55:51.316078 12290 tablet_loader.cc:96] loaded metadata for tablet 87aaf87a6c284ca980e5fb6f896c5e4e (table default.table_d0ff04fccfe8480d9d96d87f94272591 [id=fd3f033d7c714a1d860497da7452245a])
I20251025 01:55:51.316907 12290 tablet_loader.cc:96] loaded metadata for tablet 8bbdcbd2f78a49ad9dd9da514bbf7eb5 (table default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0])
I20251025 01:55:51.317174 12290 tablet_loader.cc:96] loaded metadata for tablet 937b8bb578ac4fdba8c1e800df5d6daa (table default.table_cf46809300e645fca4b99a2da934b149 [id=036e602729df45b8982a2ef049ec4e3a])
I20251025 01:55:51.318603 12290 tablet_loader.cc:96] loaded metadata for tablet a600dcbd31104aee98146b72ee257034 (table default.table_328c63794848430fb0f74a446d210d94 [id=3df0b36649ab414c9c552abbd401d14c])
I20251025 01:55:51.319192 12290 tablet_loader.cc:96] loaded metadata for tablet b3578b8a1d8f412d9a8b2b968bb9c583 (table default.table_ac99b534589f45e6871ddfa3a1331b16 [id=df0ae3390867456e81e7f7856272b2a2])
I20251025 01:55:51.319801 12290 tablet_loader.cc:96] loaded metadata for tablet c12f1f3490924e38aa7eaa2c99739803 (table default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f])
I20251025 01:55:51.320387 12290 tablet_loader.cc:96] loaded metadata for tablet ce8decc3a0fc4542bd94062e19c8fa55 (table default.table_19b5f67b2cfa4e62bc0d061b1c2c8ecf [id=43572de558b840adbf7499428842e812])
I20251025 01:55:51.320699 12290 tablet_loader.cc:96] loaded metadata for tablet d7df9d477cf1448d91cd8cac0ab321a9 (table default.table_401d389205a14ac191081b5d0df151e6 [id=69e063c1518a4234a260583bd0ba09c5])
I20251025 01:55:51.321445 12290 tablet_loader.cc:96] loaded metadata for tablet eab40c1346c54c719060421fc54f8be4 (table default.table_c18eb2691b2a4c5daf9746ee47a5e451 [id=b59ac96bfe3147de99b6697bcaaed652])
I20251025 01:55:51.321775 12290 tablet_loader.cc:96] loaded metadata for tablet effc501d8ef24438a3fd7c30feb32f87 (table default.table_b90ae6ea0cd54c2cb1ad07289907c3c2 [id=5583c1b10fd14168a2bd689a0bd5e433])
I20251025 01:55:51.322216 12290 tablet_loader.cc:96] loaded metadata for tablet fc9e67e3872b4990b0d0181eb96ad98c (table default.table_44e4ccc4f2de47198a0599e2577108cf [id=58afab4104e84c868eeccda411b8452b])
I20251025 01:55:51.322486 12290 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 01:55:51.323915 12290 catalog_manager.cc:1269] Loaded cluster ID: 21fcb89d91fc4c0cbbbcc87a8324fa65
I20251025 01:55:51.323983 12290 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 01:55:51.325609 12290 catalog_manager.cc:1514] Loading token signing keys...
I20251025 01:55:51.326905 12290 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Loaded TSK: 0
I20251025 01:55:51.327770 12290 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 01:55:51.329185 12290 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251025 01:55:51.330510 12290 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: 23
I20251025 01:55:51.402073 12093 master_service.cc:954] ReplaceTablet: received request to replace tablet 0c92525ba189465fa7eff84f166da3d0 from {username='slave'} at 127.0.0.1:41926
I20251025 01:55:51.402287 12091 master_service.cc:954] ReplaceTablet: received request to replace tablet effc501d8ef24438a3fd7c30feb32f87 from {username='slave'} at 127.0.0.1:41926
I20251025 01:55:51.402098 12092 master_service.cc:954] ReplaceTablet: received request to replace tablet f13a1ad12e2f4f2c95103cf8d76d6da6 from {username='slave'} at 127.0.0.1:41926
I20251025 01:55:51.403614 12090 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41926:
table { table_name: "default.Table_c18eb2691b2a4c5daf9746ee47a5e451" } new_table_name: "default.Table_7bca7380f82c44a4ac29fb6f6ed8a2fb" modify_external_catalogs: true
I20251025 01:55:51.404524 12089 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41926:
table { table_name: "default.Table_19b5f67b2cfa4e62bc0d061b1c2c8ecf" } modify_external_catalogs: true
I20251025 01:55:51.404448 12084 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41926:
name: "default.Table_d93bab5cc39a448caf73455685fb742f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:51.404769 12089 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41926:
table { table_name: "default.Table_19b5f67b2cfa4e62bc0d061b1c2c8ecf" } modify_external_catalogs: true
I20251025 01:55:51.403442 12086 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41926:
name: "default.Table_1a02719f7e0145b28ac84c3a43091e4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:51.405144 12092 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41926:
name: "default.Table_9391ddd403f54e168c61fa879d7f8f6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:51.406215 12087 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41926:
table { table_name: "default.Table_cf46809300e645fca4b99a2da934b149" } modify_external_catalogs: true
I20251025 01:55:51.406543 12087 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41926:
table { table_name: "default.Table_cf46809300e645fca4b99a2da934b149" } modify_external_catalogs: true
I20251025 01:55:51.406658 12088 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41926:
table { table_name: "default.Table_44e4ccc4f2de47198a0599e2577108cf" } modify_external_catalogs: true
I20251025 01:55:51.406504 12085 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41926:
name: "default.Table_356101dc020f46c490a4bbc739447d43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 01:55:51.407593 12088 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41926:
table { table_name: "default.Table_44e4ccc4f2de47198a0599e2577108cf" } modify_external_catalogs: true
F20251025 01:55:51.412284 12147 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 26, previous 26; dumping first 4 out of 13 received events: NotificationEvent(eventId=24, eventTime=1761357321, eventType=CREATE_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=25, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_a981c91dba574e1c8ccce55a41c4bd88, message={"server":"","servicePrincipal":"","db":"default","table":"table_a981c91dba574e1c8ccce55a41c4bd88","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_a981c91dba574e1c8ccce55a41c4bd88\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357320},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_a981c91dba574e1c8ccce55a41c4bd88\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_a981c91dba574e1c8ccce55a41c4bd88\",\"kudu.table_id\":\"269cfd64fbd94f0486843525917d9bb4\",\"transient_lastDdlTime\":\"1761357320\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_ac8050cb49d54bc2a444d81cf8492c75, message={"server":"","servicePrincipal":"","db":"default","table":"table_ac8050cb49d54bc2a444d81cf8492c75","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ac8050cb49d54bc2a444d81cf8492c75\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357321},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_ac8050cb49d54bc2a444d81cf8492c75\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ac8050cb49d54bc2a444d81cf8492c75\",\"kudu.table_id\":\"2f021b58f5304a3f8e7d46b745a7165a\",\"transient_lastDdlTime\":\"1761357321\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=26, eventTime=1761357321, eventType=DROP_TABLE, dbName=default, tableName=table_44e4ccc4f2de47198a0599e2577108cf, message={"server":"","servicePrincipal":"","db":"default","table":"table_44e4ccc4f2de47198a0599e2577108cf","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_44e4ccc4f2de47198a0599e2577108cf\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761357319},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/warehouse/table_44e4ccc4f2de47198a0599e2577108cf\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_44e4ccc4f2de47198a0599e2577108cf\",\"kudu.table_id\":\"58afab4104e84c868eeccda411b8452b\",\"transient_lastDdlTime\":\"1761357319\",\"comment\":\"\",\"kudu.master_addresses\":\"127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"21fcb89d91fc4c0cbbbcc87a8324fa65\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761357321}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251025 01:55:51.413725 12093 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Sending DeleteTablet for 3 replicas of tablet 0c92525ba189465fa7eff84f166da3d0
@ 0x7faf5e09cdcd google::LogMessage::Fail() at ??:0
I20251025 01:55:51.419734 12093 catalog_manager.cc:6719] ReplaceTablet: tablet 0c92525ba189465fa7eff84f166da3d0 deleted and replaced by tablet a94df48f873d4f87b33760b1c8c6c160
I20251025 01:55:51.420105 12093 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41926:
name: "default.Table_b54f3b1b46af4568a75e8141684d8f97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7faf5e0a0b93 google::LogMessage::SendToLog() at ??:0
@ 0x7faf5e09c7cc google::LogMessage::Flush() at ??:0
I20251025 01:55:51.423755 12091 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0ceaf0e1ce0f4c23b39a5bf4736de5ce: Sending DeleteTablet for 3 replicas of tablet effc501d8ef24438a3fd7c30feb32f87
I20251025 01:55:51.424304 12091 catalog_manager.cc:6719] ReplaceTablet: tablet effc501d8ef24438a3fd7c30feb32f87 deleted and replaced by tablet b9ee15faa3cc40ab8007c32701f9c3d3
I20251025 01:55:51.424703 12091 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41926:
name: "default.Table_50c0606a959945b3a1edd9df467a65dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7faf5e09ca11 google::LogMessage::~LogMessage() at ??:0
@ 0x7faf74a55b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251025 01:55:51.444271 10301 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c92525ba189465fa7eff84f166da3d0 with delete_type TABLET_DATA_DELETED (replaced by tablet a94df48f873d4f87b33760b1c8c6c160) from {username='slave'} at 127.0.0.1:60306
I20251025 01:55:51.444437 10300 tablet_service.cc:1552] Processing DeleteTablet for tablet effc501d8ef24438a3fd7c30feb32f87 with delete_type TABLET_DATA_DELETED (replaced by tablet b9ee15faa3cc40ab8007c32701f9c3d3) from {username='slave'} at 127.0.0.1:60306
I20251025 01:55:51.444877 10298 tablet_service.cc:1505] Processing CreateTablet for tablet 8bbdcbd2f78a49ad9dd9da514bbf7eb5 (DEFAULT_TABLE table=default.table_31d2075af6df48048aa516b629e1caa3 [id=f8c139c823a940e49fe874c9e4035ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:51.445353 10298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bbdcbd2f78a49ad9dd9da514bbf7eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:51.447228 12312 tablet_replica.cc:333] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
@ 0x7faf74a54411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251025 01:55:51.447448 12312 raft_consensus.cc:2243] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 LEADER]: Raft consensus shutting down.
I20251025 01:55:51.447515 10297 tablet_service.cc:1505] Processing CreateTablet for tablet a94df48f873d4f87b33760b1c8c6c160 (DEFAULT_TABLE table=default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:51.447873 12312 raft_consensus.cc:2272] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:51.447912 10297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a94df48f873d4f87b33760b1c8c6c160. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:51.448206 10299 tablet_service.cc:1505] Processing CreateTablet for tablet c12f1f3490924e38aa7eaa2c99739803 (DEFAULT_TABLE table=default.table_827ca3c8c36543b29dac05ab520c552a [id=57909f703ac64c74b92f656b0b67781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 01:55:51.448632 10299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12f1f3490924e38aa7eaa2c99739803. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 01:55:51.448668 12312 ts_tablet_manager.cc:1916] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7faf74a5c08c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7faf74a5bf09 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
I20251025 01:55:51.457857 12312 ts_tablet_manager.cc:1929] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:51.457984 12312 log.cc:1199] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/0c92525ba189465fa7eff84f166da3d0
I20251025 01:55:51.458527 12312 ts_tablet_manager.cc:1950] T 0c92525ba189465fa7eff84f166da3d0 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:51.459571 12078 catalog_manager.cc:4985] TS c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): tablet 0c92525ba189465fa7eff84f166da3d0 (table default.table_8553cdea51804bfa861b6c10537b99c3 [id=91fea98dbb4548a2babcdf286454d811]) successfully deleted
@ 0x7faf60e10ee9 kudu::Thread::SuperviseThread() at ??:0
I20251025 01:55:51.461576 10440 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c92525ba189465fa7eff84f166da3d0 with delete_type TABLET_DATA_DELETED (replaced by tablet a94df48f873d4f87b33760b1c8c6c160) from {username='slave'} at 127.0.0.1:49500
I20251025 01:55:51.461805 12312 tablet_replica.cc:333] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b: stopping tablet replica
@ 0x7faf644186db start_thread at ??:0
@ 0x7faf5cf4d71f clone at ??:0
I20251025 01:55:51.465929 12312 raft_consensus.cc:2243] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:51.466131 12312 raft_consensus.cc:2272] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:51.466689 12312 ts_tablet_manager.cc:1916] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:51.471230 12315 tablet_replica.cc:333] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: stopping tablet replica
I20251025 01:55:51.471640 12315 raft_consensus.cc:2243] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 01:55:51.471900 12315 raft_consensus.cc:2272] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 01:55:51.472566 12315 ts_tablet_manager.cc:1916] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 01:55:51.474571 12312 ts_tablet_manager.cc:1929] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:51.474697 12312 log.cc:1199] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-0/wal/wals/effc501d8ef24438a3fd7c30feb32f87
I20251025 01:55:51.475116 12312 ts_tablet_manager.cc:1950] T effc501d8ef24438a3fd7c30feb32f87 P c3e4f9f2253e416e8b31ea86ead23a3b: Deleting consensus metadata
I20251025 01:55:51.476696 12313 tablet_bootstrap.cc:492] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:51.478524 12313 tablet_bootstrap.cc:654] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:51.480401 12313 tablet_bootstrap.cc:492] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:51.480537 12313 ts_tablet_manager.cc:1403] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251025 01:55:51.480973 12315 ts_tablet_manager.cc:1929] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 01:55:51.481093 12315 log.cc:1199] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting WAL directory at /tmp/dist-test-tasknR9X9f/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761357302024081-9718-0/minicluster-data/ts-1/wal/wals/0c92525ba189465fa7eff84f166da3d0
I20251025 01:55:51.481564 12315 ts_tablet_manager.cc:1950] T 0c92525ba189465fa7eff84f166da3d0 P 1f6e00cb735e4cb2bd1221cdcbc3c62c: Deleting consensus metadata
I20251025 01:55:51.481338 12313 raft_consensus.cc:359] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:51.481691 12313 raft_consensus.cc:385] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:51.481792 12313 raft_consensus.cc:740] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:51.482007 12313 consensus_queue.cc:260] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:51.484323 12313 ts_tablet_manager.cc:1434] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 01:55:51.484758 12313 tablet_bootstrap.cc:492] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:51.486296 12313 tablet_bootstrap.cc:654] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:51.488026 12313 tablet_bootstrap.cc:492] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:51.488166 12313 ts_tablet_manager.cc:1403] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20251025 01:55:51.488893 12313 raft_consensus.cc:359] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:51.489053 12313 raft_consensus.cc:385] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:51.489143 12313 raft_consensus.cc:740] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:51.489384 12313 consensus_queue.cc:260] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:51.489856 12313 ts_tablet_manager.cc:1434] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 01:55:51.490124 12313 tablet_bootstrap.cc:492] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b: Bootstrap starting.
I20251025 01:55:51.491835 12313 tablet_bootstrap.cc:654] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b: Neither blocks nor log segments found. Creating new log.
I20251025 01:55:51.493476 12313 tablet_bootstrap.cc:492] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b: No bootstrap required, opened a new log
I20251025 01:55:51.493557 12313 ts_tablet_manager.cc:1403] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251025 01:55:51.494004 12313 raft_consensus.cc:359] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:51.494094 12313 raft_consensus.cc:385] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 01:55:51.494139 12313 raft_consensus.cc:740] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e4f9f2253e416e8b31ea86ead23a3b, State: Initialized, Role: FOLLOWER
I20251025 01:55:51.494278 12313 consensus_queue.cc:260] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:51.494570 12313 ts_tablet_manager.cc:1434] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20251025 01:55:51.568126 10372 tablet.cc:2378] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251025 01:55:51.628615 10396 connection.cc:537] server connection from 127.0.0.1:49500 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:51.628644 10256 connection.cc:537] server connection from 127.0.0.1:60306 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:51.628705 12190 connection.cc:537] server connection from 127.0.0.1:49706 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:51.629197 10256 connection.cc:537] client connection to 127.9.125.189:45553 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:51.629761 12311 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 01:55:51.416571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.9.125.131:41021)
1025 01:55:51.439435 (+ 22864us) server_negotiation.cc:207] Beginning negotiation
1025 01:55:51.439444 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 01:55:51.439466 (+ 22us) server_negotiation.cc:408] Connection header received
1025 01:55:51.439549 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 01:55:51.439557 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 01:55:51.439647 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 01:55:51.439770 (+ 123us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 01:55:51.628739 (+188969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36404: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":22739,"thread_start_us":22599,"threads_started":1}
W20251025 01:55:51.630120 10651 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.125.189:45553)
W20251025 01:55:51.630160 10397 connection.cc:537] client connection to 127.9.125.189:45553 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:51.630398 10651 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.125.189:45553)
W20251025 01:55:51.630780 10651 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.125.189:45553)
W20251025 01:55:51.630896 10533 connection.cc:537] client connection to 127.9.125.189:45553 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 01:55:51.631220 12293 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 01:55:51.630056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.125.189:45553 (local address 127.9.125.129:38225)
1025 01:55:51.630337 (+ 281us) negotiation.cc:107] Waiting for socket to connect
1025 01:55:51.630358 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1025 01:55:51.630487 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 01:55:51.630939 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":174}
W20251025 01:55:51.632063 10651 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.125.189:45553)
I20251025 01:55:51.748355 12168 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:51.748546 12168 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:51.749559 12168 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:55:51.750308 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:55:51.750494 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:55:51.750662 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:55:51.750887 12168 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:55:51.890561 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 1: this message will repeat every 5th retry.
I20251025 01:55:51.960280 12333 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:55:51.960440 12333 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:51.961119 12333 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:51.963133 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:51.963364 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:51.963450 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:51.963687 12333 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:55:52.095755 12168 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:52.095920 12168 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:52.096541 12168 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:55:52.097138 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:55:52.097306 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:55:52.097419 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:55:52.097630 12168 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:55:52.214654 12168 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:52.214879 12168 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:52.215559 12168 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:55:52.216356 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:55:52.216517 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:55:52.216629 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:55:52.216838 12168 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:55:52.500319 12339 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:55:52.500485 12339 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:52.501137 12339 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:52.503432 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:52.503631 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:52.503720 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:52.503926 12339 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:55:53.525297 12350 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:55:53.525548 12350 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:53.526189 12350 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:53.528333 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:53.528548 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:53.528614 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:53.528836 12350 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:55:53.696763 12347 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:55:53.696944 12347 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:53.697610 12347 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:55:53.698354 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:55:53.698506 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:55:53.698647 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:55:53.698866 12347 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:55:53.718201 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:55:53.718359 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:54.436968 12358 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:55:54.437145 12358 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:54.437719 12358 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:54.438798 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20251025 01:55:54.440238 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:54.440439 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:54.440523 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:54.440802 12358 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:55:55.285458 12368 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:55:55.285724 12368 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:55.286373 12368 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:55.292243 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:55.292454 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:55.292565 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:55.292860 12368 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:55:56.070611 12377 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:56.070828 12377 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:55:56.071556 12377 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:55:56.072480 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:55:56.072639 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:55:56.072793 10255 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:55:56.073061 12377 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:55:56.096941 12377 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:56.097136 12377 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:56.097724 12377 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:55:56.098603 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:55:56.098775 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:55:56.098907 10255 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:55:56.099103 12377 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:55:56.605482 12377 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:56.605706 12377 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:56.606377 12377 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:55:56.607196 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:55:56.607414 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:55:56.607532 10255 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:55:56.607756 12377 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:55:56.769367 12347 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:55:56.769546 12347 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:55:56.770153 12347 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:55:56.771071 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:55:56.771229 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:55:56.771379 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:55:56.771632 12347 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:55:57.528899 12387 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:55:57.529071 12387 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:55:57.529624 12387 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:55:57.531744 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:55:57.531937 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:55:57.532018 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:55:57.532202 12387 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:55:58.719587 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:55:58.719756 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:55:59.940625 12401 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:55:59.940824 12401 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:55:59.941547 12401 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:55:59.942575 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:55:59.942745 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:55:59.942855 10255 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:55:59.943084 12401 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:00.075961 12401 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:00.076156 12401 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:56:00.076871 12401 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:00.077854 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:56:00.078014 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:56:00.078189 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:00.078500 12401 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:00.964543 12400 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:56:00.964737 12400 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:56:00.965384 12400 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:00.966356 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:56:00.966490 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:56:00.966660 10532 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:56:00.966914 12400 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 01:56:01.163712 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 6: this message will repeat every 5th retry.
I20251025 01:56:02.117559 12414 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:02.117762 12414 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:56:02.118656 12414 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:02.119565 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:56:02.119810 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:56:02.119959 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:02.120388 12414 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:03.099711 12426 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:56:03.099902 12426 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:56:03.100558 12426 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:56:03.101626 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20251025 01:56:03.102861 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:56:03.103062 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:56:03.103174 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:56:03.103490 12426 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:56:03.721028 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:56:03.721237 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:56:04.058121 12423 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:56:04.058331 12423 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:56:04.059067 12423 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:04.060040 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:56:04.060211 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:56:04.060359 10532 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:56:04.060636 12423 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:56:04.456926 12414 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:04.457132 12414 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:56:04.457809 12414 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:04.458709 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:56:04.458909 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:56:04.459039 10255 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:04.459308 12414 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:08.668052 12414 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:08.668256 12414 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:56:08.669157 12414 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:08.670169 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:56:08.670369 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:56:08.670533 10255 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:08.670792 12414 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:08.722955 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:56:08.723202 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:56:08.802256 12468 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:56:08.802498 12468 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:56:08.803493 12468 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:56:08.804873 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251025 01:56:08.806332 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:56:08.806565 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:56:08.806654 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:56:08.806891 12468 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:56:10.483042 12414 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:10.483305 12414 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:56:10.483978 12414 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:10.484834 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:56:10.484982 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:56:10.485143 10255 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:10.485554 12414 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:10.653854 12461 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:56:10.654072 12461 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:56:10.654920 12461 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:10.655956 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:56:10.656178 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:56:10.656347 10532 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:56:10.656584 12461 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 01:56:11.765734 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 11: this message will repeat every 5th retry.
I20251025 01:56:12.359011 12414 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:12.359215 12414 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:56:12.359906 12414 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:12.360793 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:56:12.360961 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:56:12.361079 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:12.361322 12414 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:12.594307 12414 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:12.594530 12414 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:56:12.595149 12414 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:12.595999 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:56:12.596174 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:56:12.596314 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:12.596565 12414 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:13.724560 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:56:13.724769 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:56:17.923542 12510 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:17.923753 12510 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:56:17.924448 12510 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:17.925513 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:56:17.925707 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:56:17.925865 10255 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:17.926164 12510 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:56:18.055714 10370 heartbeater.cc:646] Failed to heartbeat to 127.9.125.190:35925 (75 consecutive failures): Network error: Failed to ping master at 127.9.125.190:35925: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:56:18.057029 10509 heartbeater.cc:646] Failed to heartbeat to 127.9.125.189:45553 (66 consecutive failures): Network error: Failed to ping master at 127.9.125.189:45553: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:56:18.131594 10644 heartbeater.cc:646] Failed to heartbeat to 127.9.125.190:35925 (70 consecutive failures): Network error: Failed to ping master at 127.9.125.190:35925: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
I20251025 01:56:18.726121 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:56:18.726328 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:56:19.827808 12525 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:19.828022 12525 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:56:19.828681 12525 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:19.829526 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:56:19.829742 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:56:19.829890 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:19.830108 12525 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:21.282784 12549 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:56:21.282997 12549 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:56:21.283818 12549 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:56:21.284955 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251025 01:56:21.288547 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:56:21.288751 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:56:21.288834 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:56:21.289085 12549 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20251025 01:56:21.662179 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 16: this message will repeat every 5th retry.
I20251025 01:56:23.727977 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:56:23.728193 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:56:24.421703 12571 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:24.421862 12571 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:56:24.422572 12571 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:24.423295 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:56:24.423453 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:56:24.423549 10255 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:24.423758 12571 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:25.692241 12540 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:56:25.692476 12540 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:56:25.693401 12540 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:25.694378 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:56:25.694552 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:56:25.694681 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:56:25.694965 12540 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:56:28.729645 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:56:28.729871 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:56:29.632774 12580 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:29.632999 12580 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:56:29.633697 12580 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:29.634618 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:56:29.634814 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:56:29.634971 10255 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:29.635217 12580 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:56:31.765185 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 21: this message will repeat every 5th retry.
I20251025 01:56:32.568550 12623 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:32.568782 12623 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:56:32.569401 12623 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:32.570389 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:56:32.570569 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:56:32.570672 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:32.570909 12623 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:33.732707 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:56:33.732964 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:56:34.156271 12644 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:34.156499 12644 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:56:34.157308 12644 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:34.158226 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:56:34.158535 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:56:34.158659 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:34.158941 12644 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:37.499547 12638 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:56:37.499760 12638 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:56:37.500380 12638 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:37.501209 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:56:37.501377 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:56:37.501505 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:56:37.501792 12638 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:56:37.557498 12667 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:56:37.557729 12667 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:56:37.558734 12667 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:56:37.559921 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251025 01:56:37.561252 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:56:37.561470 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:56:37.561578 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:56:37.561873 12667 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:56:38.734254 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:56:38.734455 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:56:39.204787 12664 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:39.204975 12664 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:56:39.205629 12664 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:39.206429 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:56:39.206586 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:56:39.206717 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:39.206919 12664 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:40.990044 12664 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:40.990262 12664 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:56:40.991101 12664 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:40.992102 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:56:40.992259 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:56:40.992391 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:40.992771 12664 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:56:41.826030 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 26: this message will repeat every 5th retry.
I20251025 01:56:42.041559 12704 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:56:42.041765 12704 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:56:42.042327 12704 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:56:42.044615 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:56:42.044773 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:56:42.044916 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:56:42.045126 12704 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:56:42.064663 12696 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:56:42.064886 12696 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:56:42.065716 12696 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:42.066473 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:56:42.066632 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:56:42.066766 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:56:42.067020 12696 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:56:43.736506 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:56:43.736723 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:56:48.738229 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:56:48.738405 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:56:50.108024 12764 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:56:50.108245 12764 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:56:50.108827 12764 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:56:50.109854 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251025 01:56:50.112260 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:56:50.112478 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:56:50.112623 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:56:50.112856 12764 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:56:51.712749 12763 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:51.712939 12763 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:56:51.713488 12763 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:51.714260 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:56:51.714416 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:56:51.714522 10255 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:51.714751 12763 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:56:52.316838 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 31: this message will repeat every 5th retry.
I20251025 01:56:53.739813 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:56:53.739992 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:56:53.761509 12785 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:53.761658 12785 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:56:53.762274 12785 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:53.763226 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:56:53.763425 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:56:53.763505 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:53.763695 12785 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:54.942477 12785 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:56:54.942706 12785 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:56:54.943523 12785 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:54.944429 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:56:54.944626 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:56:54.944748 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:56:54.944989 12785 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:56:56.593336 12818 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:56:56.593560 12818 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:56:56.594328 12818 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:56:56.595332 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:56:56.595504 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:56:56.595602 10532 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:56:56.595883 12818 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:56:58.741366 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:56:58.741533 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:57:01.458724 12817 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:01.458911 12817 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:57:01.459775 12817 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:01.460741 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:57:01.460923 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:57:01.461035 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:01.461265 12817 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:57:01.878712 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 36: this message will repeat every 5th retry.
I20251025 01:57:03.743021 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:57:03.743213 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:57:03.901245 12817 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:03.901422 12817 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:57:03.902047 12817 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:03.902899 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:57:03.903060 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:57:03.903190 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:03.903427 12817 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:57:04.418678 12872 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:57:04.418941 12872 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:57:04.419790 12872 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:04.420615 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:57:04.420794 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:57:04.420909 10532 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:57:04.421141 12872 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:57:04.785669 12879 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:57:04.785837 12879 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:57:04.786360 12879 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:57:04.787333 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251025 01:57:04.788561 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:57:04.788717 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:57:04.788811 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:57:04.789067 12879 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:57:08.744613 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:57:08.744822 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251025 01:57:11.515537 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 41: this message will repeat every 5th retry.
I20251025 01:57:12.231208 12926 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:12.231439 12926 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:57:12.232072 12926 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:12.233075 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:57:12.233266 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:57:12.233373 10255 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:12.233639 12926 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:57:12.819775 12931 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:12.819932 12931 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:57:12.820492 12931 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:12.821193 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:57:12.821339 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:57:12.821440 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:12.821653 12931 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:57:13.695780 12914 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:57:13.695967 12914 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:57:13.696753 12914 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:13.697652 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:57:13.697824 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:57:13.697925 10532 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:57:13.698161 12914 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:57:13.746393 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:57:13.746625 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:57:16.148025 12939 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:16.148234 12939 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:57:16.149111 12939 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:16.149825 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:57:16.150322 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:57:16.150460 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:16.150768 12939 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:57:17.244688 12939 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:17.244863 12939 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:57:17.245675 12939 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:17.246529 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:57:17.246693 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:57:17.246802 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:17.247022 12939 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:57:18.143118 10509 heartbeater.cc:646] Failed to heartbeat to 127.9.125.189:45553 (217 consecutive failures): Network error: Failed to ping master at 127.9.125.189:45553: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:57:18.187572 10370 heartbeater.cc:646] Failed to heartbeat to 127.9.125.190:35925 (195 consecutive failures): Network error: Failed to ping master at 127.9.125.190:35925: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:57:18.263041 10644 heartbeater.cc:646] Failed to heartbeat to 127.9.125.190:35925 (190 consecutive failures): Network error: Failed to ping master at 127.9.125.190:35925: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
I20251025 01:57:18.748152 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:57:18.748358 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:57:22.325685 12993 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:22.325870 12993 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:57:22.326505 12993 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:22.327276 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:57:22.327418 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:57:22.327528 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:22.327745 12993 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:57:22.399317 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 46: this message will repeat every 5th retry.
I20251025 01:57:22.422921 12993 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:22.423135 12993 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:57:22.423858 12993 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:22.424717 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:57:22.424886 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:57:22.425073 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:22.425385 12993 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:57:23.594048 12993 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:23.594255 12993 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:57:23.594931 12993 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:23.595868 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:57:23.596067 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:57:23.596199 10255 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:23.596457 12993 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:57:23.749696 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:57:23.749886 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:57:24.687333 13010 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:57:24.687543 13010 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:57:24.688367 13010 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:57:24.689541 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251025 01:57:24.691123 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:57:24.691362 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:57:24.691465 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:57:24.691725 13010 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:57:28.751313 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:57:28.751547 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:57:29.306794 12993 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:29.306977 12993 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:57:29.307646 12993 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:29.308509 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:57:29.308673 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:57:29.308786 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:29.309000 12993 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:57:31.269886 12993 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:31.270079 12993 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:57:31.270699 12993 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:31.271571 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:57:31.271744 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:57:31.271952 10255 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:31.272184 12993 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:57:31.895959 13024 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:57:31.896157 13024 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:57:31.896953 13024 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:31.897703 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:57:31.897878 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:57:31.897984 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:57:31.898200 13024 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 01:57:32.324779 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 51: this message will repeat every 5th retry.
I20251025 01:57:33.753129 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:57:33.753338 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:57:34.944242 13079 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:57:34.944473 13079 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:57:34.945083 13079 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:57:34.946146 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251025 01:57:34.947526 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:57:34.947700 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:57:34.947788 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:57:34.948009 13079 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:57:37.816990 13084 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:37.817309 13084 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:57:37.818198 13084 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:37.819108 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:57:37.819293 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:57:37.819406 10255 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:37.819641 13084 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:57:38.754642 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:57:38.754834 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251025 01:57:42.686548 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 56: this message will repeat every 5th retry.
I20251025 01:57:43.756143 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:57:43.756331 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:57:46.051759 13152 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:57:46.051949 13152 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:57:46.052632 13152 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:46.053421 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:57:46.053742 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:57:46.053881 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:57:46.054138 13152 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:57:46.662678 13151 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:46.662904 13151 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:57:46.663635 13151 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:46.664398 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:57:46.664561 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:57:46.664695 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:46.664922 13151 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:57:48.202239 13151 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:48.202414 13151 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:57:48.203024 13151 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:48.203864 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:57:48.204015 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:57:48.204144 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:48.204382 13151 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:57:48.738878 12195 maintenance_manager.cc:419] P 286b4c3025e24a7c8107763fa40d3ab1: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033344
I20251025 01:57:48.772293 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:57:48.772478 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:57:48.792104 12194 maintenance_manager.cc:643] P 286b4c3025e24a7c8107763fa40d3ab1: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.052s user 0.025s sys 0.005s Metrics: {"bytes_written":26750,"cfile_init":1,"dirs.queue_time_us":373,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":812,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":688,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":69,"thread_start_us":457,"threads_started":2,"wal-append.queue_time_us":287}
I20251025 01:57:48.793274 12195 maintenance_manager.cc:419] P 286b4c3025e24a7c8107763fa40d3ab1: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 4111 bytes on disk
I20251025 01:57:48.793882 12194 maintenance_manager.cc:643] P 286b4c3025e24a7c8107763fa40d3ab1: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20251025 01:57:51.594390 13194 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:57:51.594560 13194 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:57:51.595930 13194 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:51.596194 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:57:51.596405 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:57:51.596518 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:57:51.596781 13194 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:57:52.236423 13151 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:52.236594 13151 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:57:52.237236 13151 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:52.238098 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:57:52.238296 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:57:52.238410 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:52.238631 13151 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:57:52.240445 13151 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:57:52.240546 13151 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:57:52.240957 13151 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:52.241572 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:57:52.241714 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:57:52.241825 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:57:52.242027 13151 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:57:52.625874 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 61: this message will repeat every 5th retry.
I20251025 01:57:53.384202 13209 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:57:53.384382 13209 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:57:53.385206 13209 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:57:53.386320 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251025 01:57:53.392243 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:57:53.392448 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:57:53.392554 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:57:53.392823 13209 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:57:53.773559 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:57:53.773738 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:57:57.869867 13221 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:57:57.870065 13221 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:57:57.870656 13221 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:57:57.871469 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:57:57.871634 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:57:57.871737 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:57:57.871958 13221 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:57:58.775604 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:57:58.775851 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251025 01:58:03.157645 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 66: this message will repeat every 5th retry.
I20251025 01:58:03.777458 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:58:03.777633 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:58:06.154364 13311 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:06.154542 13311 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:58:06.155586 13311 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:06.156385 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:58:06.156539 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:58:06.156661 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:06.156901 13311 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:58:06.184389 13311 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:06.184548 13311 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:58:06.185170 13311 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:06.185897 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:58:06.186055 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:58:06.186196 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:06.186414 13311 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:58:08.124224 13352 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:58:08.124454 13352 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:58:08.125177 13352 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:58:08.126318 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251025 01:58:08.127460 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:58:08.127600 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:58:08.127681 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:58:08.127904 13352 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:58:08.762198 13311 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:08.762368 13311 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:58:08.762948 13311 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:08.763654 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:58:08.763829 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:58:08.763942 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:08.764171 13311 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:58:08.778739 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:58:08.778889 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:58:10.077888 13371 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:58:10.078104 13371 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:58:10.078958 13371 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:10.079834 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:58:10.079978 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:58:10.080108 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:58:10.080320 13371 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 01:58:13.366832 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 71: this message will repeat every 5th retry.
I20251025 01:58:13.780350 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:58:13.780530 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:58:15.052757 13417 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:15.052945 13417 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:58:15.053629 13417 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:15.054445 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:58:15.054656 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:58:15.054777 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:15.055015 13417 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:58:17.066504 13417 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:17.066728 13417 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:58:17.067646 13417 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:17.068655 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:58:17.068951 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:58:17.069036 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:17.069311 13417 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:58:18.238636 10509 heartbeater.cc:646] Failed to heartbeat to 127.9.125.189:45553 (369 consecutive failures): Network error: Failed to ping master at 127.9.125.189:45553: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 01:58:18.238659 10508 heartbeater.cc:646] Failed to heartbeat to 127.9.125.190:35925 (386 consecutive failures): Network error: Failed to ping master at 127.9.125.190:35925: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:58:18.318677 10370 heartbeater.cc:646] Failed to heartbeat to 127.9.125.190:35925 (315 consecutive failures): Network error: Failed to ping master at 127.9.125.190:35925: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:58:18.390576 10644 heartbeater.cc:646] Failed to heartbeat to 127.9.125.190:35925 (310 consecutive failures): Network error: Failed to ping master at 127.9.125.190:35925: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
I20251025 01:58:18.781996 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:58:18.782220 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:58:20.110746 13465 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:58:20.110961 13465 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:58:20.111650 13465 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:20.112522 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:58:20.112715 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:58:20.112820 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:58:20.113065 13465 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:58:22.999990 13502 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:58:23.000262 13502 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:58:23.001113 13502 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:58:23.002825 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 76: this message will repeat every 5th retry.
W20251025 01:58:23.004117 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251025 01:58:23.005328 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:58:23.005537 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:58:23.005668 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:58:23.005896 13502 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:58:23.783684 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:58:23.783874 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:58:23.823299 13501 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:23.823459 13501 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:58:23.824442 13501 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:23.825249 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:58:23.825404 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:58:23.825501 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:23.825706 13501 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:58:23.838356 13516 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:58:23.838551 13516 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:58:23.839128 13516 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:58:23.840839 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:58:23.841063 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:58:23.841159 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:58:23.841399 13516 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:58:25.648701 13532 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:58:25.648869 13532 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:58:25.649600 13532 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:25.650301 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:58:25.650478 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:58:25.650604 10532 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:58:25.650882 13532 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:58:25.825119 13501 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:25.825305 13501 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:58:25.825933 13501 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:25.826746 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:58:25.826967 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:58:25.827086 10255 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:25.827330 13501 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:58:27.079314 13553 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:58:27.079736 13553 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:58:27.080581 13553 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:58:27.082980 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:58:27.083154 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:58:27.083292 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:58:27.083547 13553 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:58:28.785216 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:58:28.785441 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:58:28.957338 13577 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:58:28.957573 13577 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:58:28.958215 13577 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:58:28.959050 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20251025 01:58:28.960292 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:58:28.960467 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:58:28.960572 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:58:28.960822 13577 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:58:29.387056 13542 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:29.387270 13542 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:58:29.387941 13542 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:29.388655 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:58:29.388804 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:58:29.388923 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:29.389135 13542 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:58:30.167515 13591 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:58:30.167707 13591 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:58:30.168254 13591 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:30.169101 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:58:30.169265 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:58:30.169394 10532 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:58:30.169642 13591 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:58:31.815035 13612 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:58:31.815233 13612 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:58:31.815893 13612 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:58:31.820305 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:58:31.820529 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:58:31.820618 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:58:31.820884 13612 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20251025 01:58:31.854874 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 81: this message will repeat every 5th retry.
I20251025 01:58:33.786841 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:58:33.787019 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:58:34.080928 13641 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:58:34.081108 13641 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:58:34.081730 13641 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:34.082648 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:58:34.082805 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:58:34.082895 10532 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:58:34.083129 13641 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:58:34.197973 13642 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:58:34.198235 13642 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:58:34.198868 13642 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:58:34.199599 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251025 01:58:34.200839 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:58:34.201040 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:58:34.201117 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:58:34.201328 13642 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:58:36.514365 13655 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:36.514549 13655 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:58:36.515120 13655 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:36.516016 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:58:36.516152 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:58:36.516289 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:36.516597 13655 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:58:36.741552 13655 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:36.741741 13655 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:58:36.742317 13655 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:36.743093 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:58:36.743288 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:58:36.743420 10255 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:36.743700 13655 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:58:38.788663 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:58:38.788851 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:58:38.839188 13692 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:58:38.839404 13692 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:58:38.840020 13692 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:58:38.841753 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:58:38.841909 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:58:38.842016 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:58:38.842259 13692 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:58:40.283160 13675 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:58:40.283329 13675 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:58:40.283867 13675 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:40.284603 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:58:40.284799 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:58:40.284916 10532 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:58:40.285140 13675 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:58:41.045904 13668 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:41.046087 13668 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:58:41.046942 13668 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:41.047853 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:58:41.048053 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:58:41.048187 10255 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:41.048434 13668 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:58:42.391232 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 86: this message will repeat every 5th retry.
I20251025 01:58:42.615893 13720 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:42.616070 13720 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:58:42.616739 13720 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:42.617558 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:58:42.617692 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:58:42.617877 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:42.618132 13720 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:58:43.790095 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:58:43.790277 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:58:47.914479 13720 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:47.914674 13720 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:58:47.915369 13720 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:47.916191 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:58:47.916376 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:58:47.916507 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:47.916752 13720 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:58:48.791831 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:58:48.792018 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251025 01:58:52.223112 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 91: this message will repeat every 5th retry.
I20251025 01:58:52.530572 13820 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:52.530747 13820 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:58:52.531445 13820 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:52.532092 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:58:52.532248 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:58:52.532366 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:52.532624 13820 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:58:53.793275 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:58:53.793481 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:58:55.601045 13836 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:55.601238 13836 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:58:55.601866 13836 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:55.602655 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:58:55.602819 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:58:55.602931 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:55.603155 13836 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:58:55.871287 13855 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:58:55.871490 13855 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:58:55.872292 13855 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:58:55.873579 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251025 01:58:55.874895 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:58:55.875077 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:58:55.875180 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:58:55.875449 13855 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:58:57.515672 13874 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:58:57.515834 13874 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:58:57.516391 13874 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:58:57.518270 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:58:57.518427 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:58:57.518498 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:58:57.518702 13874 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:58:58.536597 13836 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:58:58.536763 13836 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:58:58.537456 13836 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:58.538121 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:58:58.538262 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:58:58.538401 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:58:58.538614 13836 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:58:58.794816 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:58:58.795039 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:58:59.633220 13868 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:58:59.633445 13868 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:58:59.634341 13868 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:58:59.635213 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:58:59.635406 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:58:59.635524 10532 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:58:59.635769 13868 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:59:00.772817 13904 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:59:00.772989 13904 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:59:00.773564 13904 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:59:00.775563 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:59:00.775758 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:59:00.775882 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:59:00.776113 13904 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:59:02.129534 13836 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:02.129705 13836 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:59:02.130286 13836 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:02.131050 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:59:02.131204 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:59:02.131433 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:02.131690 13836 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:59:02.545073 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 96: this message will repeat every 5th retry.
I20251025 01:59:03.062512 13836 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:03.062690 13836 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:59:03.063345 13836 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:03.064181 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:59:03.064370 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:59:03.064497 10255 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:03.064738 13836 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:59:03.496181 13836 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:03.496393 13836 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:59:03.497044 13836 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:03.497777 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:59:03.497918 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:59:03.498039 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:03.498239 13836 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:59:03.796579 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:59:03.796785 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:59:06.989631 13965 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:59:06.989810 13965 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:59:06.990434 13965 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:06.991312 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:59:06.991487 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:59:06.991626 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:59:06.991900 13965 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:59:08.798259 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:59:08.798460 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:59:12.739614 13999 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:12.739804 13999 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:59:12.740453 13999 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:12.741199 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:59:12.741355 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:59:12.741462 10255 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:12.741747 13999 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:59:12.851014 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 101: this message will repeat every 5th retry.
I20251025 01:59:13.800021 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:59:13.800232 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:59:15.674605 14023 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:15.674790 14023 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:59:15.675576 14023 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:15.676523 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:59:15.676687 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:59:15.676812 10255 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:15.677037 14023 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:59:18.441044 10370 heartbeater.cc:646] Failed to heartbeat to 127.9.125.190:35925 (435 consecutive failures): Network error: Failed to ping master at 127.9.125.190:35925: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:59:18.517454 10644 heartbeater.cc:646] Failed to heartbeat to 127.9.125.190:35925 (430 consecutive failures): Network error: Failed to ping master at 127.9.125.190:35925: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:59:18.693439 10509 heartbeater.cc:646] Failed to heartbeat to 127.9.125.189:45553 (517 consecutive failures): Network error: Failed to ping master at 127.9.125.189:45553: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:59:18.801685 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:59:18.801931 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:59:19.043913 14023 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:19.044078 14023 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:59:19.044605 14023 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:19.045449 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:59:19.045583 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:59:19.045666 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:19.045832 14023 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:59:19.049170 14080 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:59:19.049328 14080 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:59:19.050109 14080 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:59:19.050885 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20251025 01:59:19.052282 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:59:19.052420 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:59:19.052524 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:59:19.052759 14080 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:59:21.065634 14012 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:59:21.065855 14012 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:59:21.066738 14012 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:21.067634 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:59:21.067795 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:59:21.067920 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:59:21.068194 14012 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 01:59:22.437458 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 106: this message will repeat every 5th retry.
I20251025 01:59:23.202215 14023 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:23.202391 14023 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:59:23.202911 14023 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:23.203755 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:59:23.203861 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:59:23.203995 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:23.204206 14023 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:59:23.803508 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:59:23.803689 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:59:24.203101 14023 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:24.203287 14023 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:59:24.203845 14023 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:24.204800 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:59:24.204963 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:59:24.205086 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:24.205320 14023 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:59:27.715037 14144 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:27.715201 14144 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:59:27.715701 14144 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:27.716516 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:59:27.716640 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:59:27.716743 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:27.716933 14144 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:59:28.805019 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:59:28.805220 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:59:31.363442 14191 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:59:31.363654 14191 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:59:31.364315 14191 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:31.365290 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:59:31.365474 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:59:31.365558 10532 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:59:31.365772 14191 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 01:59:32.000705 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 111: this message will repeat every 5th retry.
I20251025 01:59:32.163336 14144 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:32.163522 14144 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:59:32.164415 14144 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:32.165216 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:59:32.165434 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:59:32.165535 10255 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:32.165762 14144 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:59:33.313076 14144 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:33.313274 14144 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:59:33.313843 14144 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:33.314751 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:59:33.314884 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:59:33.315021 10255 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:33.315289 14144 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:59:33.806720 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:59:33.806917 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:59:38.713608 14269 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:59:38.713838 14269 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:59:38.714779 14269 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:59:38.716271 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251025 01:59:38.717675 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:59:38.717849 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:59:38.717954 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:59:38.718192 14269 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:59:38.807931 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:59:38.808132 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:59:39.583966 14220 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:39.584162 14220 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:59:39.584743 14220 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:39.585721 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:59:39.585852 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:59:39.585975 10254 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:39.586169 14220 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:59:39.653456 14284 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:59:39.653640 14284 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:59:39.654300 14284 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:59:39.660306 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:59:39.660530 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:59:39.660616 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:59:39.660863 14284 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:59:41.280283 14304 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:59:41.280493 14304 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:59:41.281047 14304 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:59:41.284312 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:59:41.284528 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:59:41.284648 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:59:41.284942 14304 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20251025 01:59:42.759346 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 116: this message will repeat every 5th retry.
I20251025 01:59:43.809758 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:59:43.810003 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:59:46.631532 14220 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:46.631711 14220 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:59:46.632314 14220 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:46.633292 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:59:46.633445 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:59:46.633533 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:46.633729 14220 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:59:47.978631 14220 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:47.978829 14220 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:59:47.979668 14220 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:47.980566 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:59:47.980693 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:59:47.980825 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:47.981053 14220 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:59:48.409441 14363 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:59:48.409657 14363 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:59:48.410454 14363 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:48.411466 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:59:48.411626 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:59:48.411731 10532 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:59:48.411949 14363 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:59:48.811425 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:59:48.811654 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:59:49.912911 14220 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:49.913089 14220 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:59:49.913635 14220 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:49.914491 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:59:49.914603 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:59:49.914716 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:49.914919 14220 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:59:51.688937 14220 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:51.689095 14220 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 01:59:51.689699 14220 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:51.690546 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 01:59:51.690671 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 01:59:51.690779 10255 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:51.690971 14220 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 01:59:52.662288 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 121: this message will repeat every 5th retry.
I20251025 01:59:53.245782 14413 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:59:53.246011 14413 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:59:53.246783 14413 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:53.247722 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:59:53.247882 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:59:53.247994 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:59:53.248221 14413 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:59:53.813378 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:59:53.813577 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 01:59:54.453418 14419 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:54.453590 14419 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:59:54.454131 14419 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:54.454975 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 01:59:54.455118 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 01:59:54.455236 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:54.455493 14419 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:59:55.276266 14419 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 01:59:55.276453 14419 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 01:59:55.277045 14419 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:55.277801 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 01:59:55.277915 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 01:59:55.278052 10255 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 01:59:55.278338 14419 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 01:59:56.569919 14430 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 01:59:56.570073 14430 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 01:59:56.570711 14430 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 01:59:56.571470 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 01:59:56.571612 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 01:59:56.571734 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 01:59:56.571964 14430 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 01:59:57.095425 14455 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 01:59:57.095657 14455 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 01:59:57.096593 14455 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 01:59:57.097839 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20251025 01:59:57.100344 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 01:59:57.100560 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 01:59:57.100652 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 01:59:57.100939 14455 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 01:59:58.815038 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 01:59:58.815233 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 02:00:00.678121 14460 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 02:00:00.678321 14460 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 02:00:00.678905 14460 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 02:00:00.679857 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 02:00:00.679997 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 02:00:00.680126 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 02:00:00.680367 14460 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 02:00:02.175836 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 126: this message will repeat every 5th retry.
I20251025 02:00:03.817130 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 02:00:03.817358 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 02:00:08.067864 14557 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 02:00:08.068061 14557 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 02:00:08.068928 14557 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 02:00:08.069924 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 02:00:08.070068 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 02:00:08.070191 10255 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 02:00:08.070427 14557 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 02:00:08.818861 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 02:00:08.819077 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 02:00:09.653821 14557 raft_consensus.cc:493] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 02:00:09.653996 14557 raft_consensus.cc:515] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } }
I20251025 02:00:09.654608 14557 leader_election.cc:290] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 02:00:09.655485 10254 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
W20251025 02:00:09.655618 10255 leader_election.cc:343] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: 8bbdcbd2f78a49ad9dd9da514bbf7eb5
I20251025 02:00:09.655702 10255 leader_election.cc:304] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 02:00:09.655892 14557 raft_consensus.cc:2749] T 8bbdcbd2f78a49ad9dd9da514bbf7eb5 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 02:00:09.709268 14579 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 02:00:09.709431 14579 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 02:00:09.710227 14579 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 02:00:09.711119 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 02:00:09.711238 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 02:00:09.711375 10532 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 02:00:09.711592 14579 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 02:00:09.957386 14557 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 02:00:09.957561 14557 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 02:00:09.958163 14557 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 02:00:09.959084 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 02:00:09.959231 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 02:00:09.959375 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 02:00:09.959590 14557 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 02:00:11.374114 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 131: this message will repeat every 5th retry.
I20251025 02:00:13.821179 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 02:00:13.821380 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 02:00:15.603681 14638 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 02:00:15.603842 14638 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 02:00:15.604619 14638 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 02:00:15.606662 12192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251025 02:00:15.608103 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 02:00:15.608323 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 02:00:15.608434 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 02:00:15.608703 14638 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 02:00:15.984167 14644 raft_consensus.cc:493] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3e4f9f2253e416e8b31ea86ead23a3b)
I20251025 02:00:15.984383 14644 raft_consensus.cc:515] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } }
I20251025 02:00:15.984961 14644 leader_election.cc:290] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 02:00:15.985785 10532 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
W20251025 02:00:15.985925 10531 leader_election.cc:343] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: 0c92525ba189465fa7eff84f166da3d0
I20251025 02:00:15.986042 10531 leader_election.cc:304] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90109d2fdb464942b20891b5856f8ac2; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, c3e4f9f2253e416e8b31ea86ead23a3b
I20251025 02:00:15.986310 14644 raft_consensus.cc:2749] T 0c92525ba189465fa7eff84f166da3d0 P 90109d2fdb464942b20891b5856f8ac2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 02:00:17.282573 14557 raft_consensus.cc:493] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 02:00:17.282747 14557 raft_consensus.cc:515] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 02:00:17.283309 14557 leader_election.cc:290] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 02:00:17.284220 10255 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
W20251025 02:00:17.284385 10254 leader_election.cc:343] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: c12f1f3490924e38aa7eaa2c99739803
I20251025 02:00:17.284494 10254 leader_election.cc:304] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 02:00:17.284680 14557 raft_consensus.cc:2749] T c12f1f3490924e38aa7eaa2c99739803 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 02:00:17.285521 14657 raft_consensus.cc:493] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ceaf0e1ce0f4c23b39a5bf4736de5ce)
I20251025 02:00:17.285727 14657 raft_consensus.cc:515] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76559300dae44c0d8a17b583a8875b3f" member_type: VOTER last_known_addr { host: "127.9.125.190" port: 35925 } } peers { permanent_uuid: "0ceaf0e1ce0f4c23b39a5bf4736de5ce" member_type: VOTER last_known_addr { host: "127.9.125.189" port: 45553 } } peers { permanent_uuid: "286b4c3025e24a7c8107763fa40d3ab1" member_type: VOTER last_known_addr { host: "127.9.125.188" port: 36641 } }
I20251025 02:00:17.286257 14657 leader_election.cc:290] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925), 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553)
W20251025 02:00:17.288568 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 76559300dae44c0d8a17b583a8875b3f (127.9.125.190:35925): Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 02:00:17.288786 12192 leader_election.cc:336] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0ceaf0e1ce0f4c23b39a5bf4736de5ce (127.9.125.189:45553): Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
I20251025 02:00:17.288882 12192 leader_election.cc:304] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286b4c3025e24a7c8107763fa40d3ab1; no voters: 0ceaf0e1ce0f4c23b39a5bf4736de5ce, 76559300dae44c0d8a17b583a8875b3f
I20251025 02:00:17.289108 14657 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 286b4c3025e24a7c8107763fa40d3ab1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251025 02:00:18.456472 14557 raft_consensus.cc:493] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 02:00:18.456656 14557 raft_consensus.cc:515] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90109d2fdb464942b20891b5856f8ac2" member_type: VOTER last_known_addr { host: "127.9.125.131" port: 41021 } } peers { permanent_uuid: "1f6e00cb735e4cb2bd1221cdcbc3c62c" member_type: VOTER last_known_addr { host: "127.9.125.130" port: 43549 } } peers { permanent_uuid: "c3e4f9f2253e416e8b31ea86ead23a3b" member_type: VOTER last_known_addr { host: "127.9.125.129" port: 44581 } }
I20251025 02:00:18.457297 14557 leader_election.cc:290] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021), 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549)
W20251025 02:00:18.458298 10255 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90109d2fdb464942b20891b5856f8ac2 (127.9.125.131:41021): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
W20251025 02:00:18.458410 10254 leader_election.cc:343] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f6e00cb735e4cb2bd1221cdcbc3c62c (127.9.125.130:43549): Not found: Tablet not found: a94df48f873d4f87b33760b1c8c6c160
I20251025 02:00:18.458638 10254 leader_election.cc:304] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3e4f9f2253e416e8b31ea86ead23a3b; no voters: 1f6e00cb735e4cb2bd1221cdcbc3c62c, 90109d2fdb464942b20891b5856f8ac2
I20251025 02:00:18.458899 14557 raft_consensus.cc:2749] T a94df48f873d4f87b33760b1c8c6c160 P c3e4f9f2253e416e8b31ea86ead23a3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 02:00:18.568885 10370 heartbeater.cc:646] Failed to heartbeat to 127.9.125.190:35925 (555 consecutive failures): Network error: Failed to ping master at 127.9.125.190:35925: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 02:00:18.651774 10644 heartbeater.cc:646] Failed to heartbeat to 127.9.125.190:35925 (550 consecutive failures): Network error: Failed to ping master at 127.9.125.190:35925: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
W20251025 02:00:18.775839 10508 heartbeater.cc:646] Failed to heartbeat to 127.9.125.190:35925 (682 consecutive failures): Network error: Failed to ping master at 127.9.125.190:35925: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111)
I20251025 02:00:18.822767 12170 master_runner.cc:186] Error getting master registration for 127.9.125.190:35925: Network error: Client connection negotiation failed: client connection to 127.9.125.190:35925: connect: Connection refused (error 111),
I20251025 02:00:18.822954 12170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251025 02:00:21.334337 10396 consensus_peers.cc:597] T effc501d8ef24438a3fd7c30feb32f87 P 1f6e00cb735e4cb2bd1221cdcbc3c62c -> Peer c3e4f9f2253e416e8b31ea86ead23a3b (127.9.125.129:44581): Couldn't send request to peer c3e4f9f2253e416e8b31ea86ead23a3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: effc501d8ef24438a3fd7c30feb32f87. This is attempt 136: this message will repeat every 5th retry.
W20251025 02:00:21.567965 10651 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 02:00:21.568261 10651 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 02:00:21.568382 10651 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
W20251025 02:00:21.568563 10651 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
F20251025 02:00:21.580168 10669 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1761357621 (unix time) try "date -d @1761357621" if you are using GNU date ***
W20251025 02:00:21.583748 10651 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.125.189:45553)
F20251025 02:00:21.585940 10672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251025 02:00:21.585940 10672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586213 10663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20c488ac6bf24db48ea2ab061d253e7c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251025 02:00:21.585940 10672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586213 10663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20c488ac6bf24db48ea2ab061d253e7c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586258 10671 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251025 02:00:21.585940 10672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586213 10663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20c488ac6bf24db48ea2ab061d253e7c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586258 10671 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587306 10670 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251025 02:00:21.585940 10672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586213 10663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20c488ac6bf24db48ea2ab061d253e7c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586258 10671 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587306 10670 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587314 10662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac8050cb49d54bc2a444d81cf8492c75 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251025 02:00:21.585940 10672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586213 10663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20c488ac6bf24db48ea2ab061d253e7c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586258 10671 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587306 10670 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587314 10662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac8050cb49d54bc2a444d81cf8492c75 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587692 10673 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251025 02:00:21.585940 10672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586213 10663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20c488ac6bf24db48ea2ab061d253e7c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586258 10671 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587306 10670 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587314 10662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac8050cb49d54bc2a444d81cf8492c75 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587692 10673 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587990 10675 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251025 02:00:21.585940 10672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586213 10663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20c488ac6bf24db48ea2ab061d253e7c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586258 10671 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587306 10670 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587314 10662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac8050cb49d54bc2a444d81cf8492c75 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587692 10673 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587990 10675 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.588038 10668 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
PC: @ 0x0 (unknown)
*** Check failure stack trace: ***
F20251025 02:00:21.585940 10672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586213 10663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20c488ac6bf24db48ea2ab061d253e7c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586258 10671 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587306 10670 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587314 10662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac8050cb49d54bc2a444d81cf8492c75 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587692 10673 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587990 10675 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.588038 10668 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.588203 10676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
*** Check failure stack trace: ***
W20251025 02:00:21.588315 10651 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.125.189:45553)
*** SIGABRT (@0x3e8000025f6) received by PID 9718 (TID 0x7fc1f74a8700) from PID 9718; stack trace: ***
F20251025 02:00:21.585940 10672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586213 10663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20c488ac6bf24db48ea2ab061d253e7c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586258 10671 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587306 10670 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587314 10662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac8050cb49d54bc2a444d81cf8492c75 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587692 10673 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587990 10675 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.588038 10668 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.588203 10676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.589251 10667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251025 02:00:21.585940 10672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586213 10663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20c488ac6bf24db48ea2ab061d253e7c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586258 10671 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587306 10670 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587314 10662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac8050cb49d54bc2a444d81cf8492c75 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587692 10673 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587990 10675 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.588038 10668 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.588203 10676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.589251 10667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.589534 10674 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251025 02:00:21.585940 10672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586213 10663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20c488ac6bf24db48ea2ab061d253e7c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.586258 10671 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587306 10670 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587314 10662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ac8050cb49d54bc2a444d81cf8492c75 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587692 10673 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.587990 10675 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.588038 10668 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.588203 10676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.589251 10667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.589534 10674 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)F20251025 02:00:21.589666 10660 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a981c91dba574e1c8ccce55a41c4bd88 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.125.190:35925,127.9.125.189:45553,127.9.125.188:36641, num_attempts: 731) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.125.189:45553: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fc210064980 (unknown) at ??:0
@ 0x7fc208e3dfb7 gsignal at ??:0
@ 0x7fc208e3f921 abort at ??:0
@ 0x7fc20bb75dcd google::LogMessage::Fail() at ??:0
@ 0x7fc20bb79b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fc20bb757cc google::LogMessage::Flush() at ??:0
@ 0x7fc20bb76f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40c481 kudu::MasterStressTest::AlterTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:328
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7fc2094c56df (unknown) at ??:0
@ 0x7fc2100596db start_thread at ??:0
@ 0x7fc208f2071f clone at ??:0