Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
2026-02-04T02:53:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-04T02:53:16Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task6hEvLx/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task6hEvLx/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/security.properties
2026-02-04 02:53:22: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task6hEvLx/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task6hEvLx/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260204 02:53:29.903709 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.190:33671
--webserver_interface=127.0.104.190
--webserver_port=0
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260204 02:53:30.225601 659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:53:30.225934 659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:53:30.226056 659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260204 02:53:30.235530 659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260204 02:53:30.235651 659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:53:30.235761 659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260204 02:53:30.235834 659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260204 02:53:30.245442 659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.190:33671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:53:30.247113 659 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:53:30.248996 659 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260204 02:53:30.256309 664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:30.256335 665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:30.257869 667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:53:30.260303 659 server_base.cc:1060] running on GCE node
I20260204 02:53:30.261022 659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:53:30.262326 659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:53:30.263567 659 hybrid_clock.cc:648] HybridClock initialized: now 1770173610263510 us; error 58 us; skew 500 ppm
I20260204 02:53:30.263875 659 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:53:30.265954 659 webserver.cc:492] Webserver started at http://127.0.104.190:46841/ using document root <none> and password file <none>
I20260204 02:53:30.266667 659 fs_manager.cc:362] Metadata directory not provided
I20260204 02:53:30.266799 659 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:53:30.267163 659 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260204 02:53:30.269392 659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data/instance:
uuid: "3e85e1dc6a5d4568ad365ba1d77398ff"
format_stamp: "Formatted at 2026-02-04 02:53:30 on dist-test-slave-xxhz"
I20260204 02:53:30.270093 659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/wal/instance:
uuid: "3e85e1dc6a5d4568ad365ba1d77398ff"
format_stamp: "Formatted at 2026-02-04 02:53:30 on dist-test-slave-xxhz"
I20260204 02:53:30.275027 659 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260204 02:53:30.278326 673 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:30.279742 659 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260204 02:53:30.279903 659 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/wal
uuid: "3e85e1dc6a5d4568ad365ba1d77398ff"
format_stamp: "Formatted at 2026-02-04 02:53:30 on dist-test-slave-xxhz"
I20260204 02:53:30.280107 659 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:53:30.300254 659 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:53:30.300994 659 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:53:30.301244 659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260204 02:53:30.321201 659 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.190:33671
I20260204 02:53:30.321208 724 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.190:33671 every 8 connection(s)
I20260204 02:53:30.322546 659 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data/info.pb
I20260204 02:53:30.326840 725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:30.331178 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 659
I20260204 02:53:30.331429 418 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/wal/instance
I20260204 02:53:30.332073 725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } has no permanent_uuid. Determining permanent_uuid...
I20260204 02:53:30.333827 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.189:46765
--webserver_interface=127.0.104.189
--webserver_port=0
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260204 02:53:30.351707 725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } has no permanent_uuid. Determining permanent_uuid...
W20260204 02:53:30.352718 677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:53:30.354287 725 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.189:46765: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:30.403650 725 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } attempt: 1
W20260204 02:53:30.405427 725 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.189:46765: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:30.473696 725 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } attempt: 2
W20260204 02:53:30.475513 725 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.189:46765: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:30.566802 725 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } attempt: 3
W20260204 02:53:30.568974 725 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.189:46765: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:53:30.641129 728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:53:30.641497 728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:53:30.641647 728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260204 02:53:30.650105 728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260204 02:53:30.650197 728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:53:30.650287 728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260204 02:53:30.650382 728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260204 02:53:30.659706 728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.189:46765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:53:30.661711 728 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:53:30.663429 728 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260204 02:53:30.671020 734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:30.671047 735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:30.672194 737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:53:30.672839 728 server_base.cc:1060] running on GCE node
I20260204 02:53:30.674062 728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:53:30.675428 728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:53:30.676702 728 hybrid_clock.cc:648] HybridClock initialized: now 1770173610676664 us; error 44 us; skew 500 ppm
I20260204 02:53:30.677042 728 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:53:30.678999 728 webserver.cc:492] Webserver started at http://127.0.104.189:36571/ using document root <none> and password file <none>
I20260204 02:53:30.679751 728 fs_manager.cc:362] Metadata directory not provided
I20260204 02:53:30.679888 728 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:53:30.680233 728 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260204 02:53:30.682370 728 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/instance:
uuid: "78eb37be156448e8879b0df55832ab2d"
format_stamp: "Formatted at 2026-02-04 02:53:30 on dist-test-slave-xxhz"
I20260204 02:53:30.683056 728 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal/instance:
uuid: "78eb37be156448e8879b0df55832ab2d"
format_stamp: "Formatted at 2026-02-04 02:53:30 on dist-test-slave-xxhz"
I20260204 02:53:30.688242 728 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260204 02:53:30.691195 743 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:30.692370 728 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260204 02:53:30.692533 728 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
uuid: "78eb37be156448e8879b0df55832ab2d"
format_stamp: "Formatted at 2026-02-04 02:53:30 on dist-test-slave-xxhz"
I20260204 02:53:30.692724 728 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:53:30.712222 725 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } attempt: 4
W20260204 02:53:30.713883 725 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.189:46765: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:30.721040 728 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:53:30.721860 728 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:53:30.722118 728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260204 02:53:30.742501 728 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.189:46765
I20260204 02:53:30.742522 794 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.189:46765 every 8 connection(s)
I20260204 02:53:30.743935 728 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/info.pb
I20260204 02:53:30.747766 795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:30.750483 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 728
I20260204 02:53:30.750692 418 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal/instance
I20260204 02:53:30.751760 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.188:38789
--webserver_interface=127.0.104.188
--webserver_port=0
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260204 02:53:30.753013 795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } has no permanent_uuid. Determining permanent_uuid...
I20260204 02:53:30.765714 795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } has no permanent_uuid. Determining permanent_uuid...
I20260204 02:53:30.775076 795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } has no permanent_uuid. Determining permanent_uuid...
W20260204 02:53:30.776166 744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
W20260204 02:53:30.778196 795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.188:38789: Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:30.827605 795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } attempt: 1
W20260204 02:53:30.829360 795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.188:38789: Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:30.897567 795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } attempt: 2
W20260204 02:53:30.899356 795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.188:38789: Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:30.990634 795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } attempt: 3
W20260204 02:53:30.992728 795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.188:38789: Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:31.013190 725 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } attempt: 5
I20260204 02:53:31.019699 725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } has no permanent_uuid. Determining permanent_uuid...
W20260204 02:53:31.021327 725 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.188:38789: Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:31.072566 725 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } attempt: 1
W20260204 02:53:31.074926 725 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.188:38789: Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
W20260204 02:53:31.081667 796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:53:31.082005 796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:53:31.082119 796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260204 02:53:31.090170 796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260204 02:53:31.090278 796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:53:31.090381 796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260204 02:53:31.090458 796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260204 02:53:31.099749 796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.188:38789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:53:31.101287 796 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:53:31.102844 796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260204 02:53:31.109930 805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:31.109941 804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:31.112488 807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:53:31.112998 796 server_base.cc:1060] running on GCE node
I20260204 02:53:31.113595 796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:53:31.114734 796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:53:31.115912 796 hybrid_clock.cc:648] HybridClock initialized: now 1770173611115867 us; error 37 us; skew 500 ppm
I20260204 02:53:31.116238 796 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:53:31.118654 796 webserver.cc:492] Webserver started at http://127.0.104.188:40055/ using document root <none> and password file <none>
I20260204 02:53:31.119340 796 fs_manager.cc:362] Metadata directory not provided
I20260204 02:53:31.119444 796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:53:31.119797 796 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260204 02:53:31.123108 796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/instance:
uuid: "202e4a0debb24c8aa89a4724d72fcf37"
format_stamp: "Formatted at 2026-02-04 02:53:31 on dist-test-slave-xxhz"
I20260204 02:53:31.123862 796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal/instance:
uuid: "202e4a0debb24c8aa89a4724d72fcf37"
format_stamp: "Formatted at 2026-02-04 02:53:31 on dist-test-slave-xxhz"
I20260204 02:53:31.128731 796 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260204 02:53:31.131614 813 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:31.132896 796 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260204 02:53:31.133041 796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
uuid: "202e4a0debb24c8aa89a4724d72fcf37"
format_stamp: "Formatted at 2026-02-04 02:53:31 on dist-test-slave-xxhz"
I20260204 02:53:31.133235 796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:53:31.135969 795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } attempt: 4
W20260204 02:53:31.137717 795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.188:38789: Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:31.143213 725 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } attempt: 2
W20260204 02:53:31.144846 725 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.188:38789: Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:31.146006 796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:53:31.146764 796 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:53:31.147017 796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260204 02:53:31.167065 796 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.188:38789
I20260204 02:53:31.167083 864 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.188:38789 every 8 connection(s)
I20260204 02:53:31.168480 796 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
I20260204 02:53:31.169003 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 796
I20260204 02:53:31.169178 418 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal/instance
I20260204 02:53:31.172319 865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:31.178508 865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } has no permanent_uuid. Determining permanent_uuid...
I20260204 02:53:31.190546 865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } has no permanent_uuid. Determining permanent_uuid...
I20260204 02:53:31.197575 865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } has no permanent_uuid. Determining permanent_uuid...
I20260204 02:53:31.208169 865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap starting.
I20260204 02:53:31.210866 865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:31.211818 865 log.cc:826] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Log is configured to *not* fsync() on all Append() calls
I20260204 02:53:31.214196 865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: No bootstrap required, opened a new log
I20260204 02:53:31.218750 865 raft_consensus.cc:359] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:31.218994 865 raft_consensus.cc:385] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:31.219074 865 raft_consensus.cc:740] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202e4a0debb24c8aa89a4724d72fcf37, State: Initialized, Role: FOLLOWER
I20260204 02:53:31.219638 865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:31.220317 873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:31.220561 873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:31.221218 865 sys_catalog.cc:565] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: configured and running, proceeding with master startup.
W20260204 02:53:31.233021 887 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260204 02:53:31.233217 887 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260204 02:53:31.230518 796 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: OK, instance_id { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" instance_seqno: 1770173610315721 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260204 02:53:31.238657 796 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:53:31.251045 725 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } attempt: 3
I20260204 02:53:31.263206 725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: Bootstrap starting.
I20260204 02:53:31.267198 725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:31.269780 725 log.cc:826] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: Log is configured to *not* fsync() on all Append() calls
I20260204 02:53:31.280869 725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: No bootstrap required, opened a new log
I20260204 02:53:31.290285 725 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:31.290611 725 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:31.290722 725 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e85e1dc6a5d4568ad365ba1d77398ff, State: Initialized, Role: FOLLOWER
I20260204 02:53:31.291268 725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:31.292594 891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:31.292920 891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:31.293679 725 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: configured and running, proceeding with master startup.
W20260204 02:53:31.303733 679 tablet.cc:2378] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260204 02:53:31.305856 659 master_runner.cc:186] Error getting master registration for 127.0.104.189:46765: OK, instance_id { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" instance_seqno: 1770173610736963 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260204 02:53:31.314276 659 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:53:31.318360 905 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260204 02:53:31.318507 905 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260204 02:53:31.398864 819 tablet.cc:2378] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260204 02:53:31.436944 795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } attempt: 5
I20260204 02:53:31.448637 795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Bootstrap starting.
I20260204 02:53:31.452519 795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:31.453706 795 log.cc:826] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Log is configured to *not* fsync() on all Append() calls
I20260204 02:53:31.457908 795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: No bootstrap required, opened a new log
I20260204 02:53:31.464524 795 raft_consensus.cc:359] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:31.464819 795 raft_consensus.cc:385] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:31.464975 795 raft_consensus.cc:740] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78eb37be156448e8879b0df55832ab2d, State: Initialized, Role: FOLLOWER
I20260204 02:53:31.465582 795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:31.466507 909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:31.466832 909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:31.467664 795 sys_catalog.cc:565] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: configured and running, proceeding with master startup.
W20260204 02:53:31.476071 749 tablet.cc:2378] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260204 02:53:31.487941 924 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260204 02:53:31.497438 728 master_runner.cc:263] No leader master found from master 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:31.497546 728 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:53:31.526787 891 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:31.527135 891 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:31.529076 891 leader_election.cc:290] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
I20260204 02:53:31.530133 840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202e4a0debb24c8aa89a4724d72fcf37" is_pre_election: true
I20260204 02:53:31.530088 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78eb37be156448e8879b0df55832ab2d" is_pre_election: true
I20260204 02:53:31.530570 840 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e85e1dc6a5d4568ad365ba1d77398ff in term 0.
I20260204 02:53:31.530565 770 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e85e1dc6a5d4568ad365ba1d77398ff in term 0.
I20260204 02:53:31.531142 674 leader_election.cc:304] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff; no voters:
I20260204 02:53:31.531880 891 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:31.532032 891 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:31.532148 891 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:31.535732 891 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:31.536412 891 leader_election.cc:290] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 1 election: Requested vote from peers 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
I20260204 02:53:31.536849 840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202e4a0debb24c8aa89a4724d72fcf37"
I20260204 02:53:31.536969 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78eb37be156448e8879b0df55832ab2d"
I20260204 02:53:31.537029 840 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:31.537148 770 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:31.540599 770 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e85e1dc6a5d4568ad365ba1d77398ff in term 1.
I20260204 02:53:31.540620 840 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e85e1dc6a5d4568ad365ba1d77398ff in term 1.
I20260204 02:53:31.540944 677 leader_election.cc:304] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d; no voters:
I20260204 02:53:31.541226 891 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:31.541719 891 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 1 LEADER]: Becoming Leader. State: Replica: 3e85e1dc6a5d4568ad365ba1d77398ff, State: Running, Role: LEADER
I20260204 02:53:31.542274 891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:31.546342 925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e85e1dc6a5d4568ad365ba1d77398ff. Latest consensus state: current_term: 1 leader_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:31.546810 925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: This master's current role is: LEADER
I20260204 02:53:31.547677 927 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260204 02:53:31.549785 927 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260204 02:53:31.558173 840 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 1 FOLLOWER]: Refusing update from remote peer 3e85e1dc6a5d4568ad365ba1d77398ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260204 02:53:31.558792 770 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 1 FOLLOWER]: Refusing update from remote peer 3e85e1dc6a5d4568ad365ba1d77398ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260204 02:53:31.559760 891 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260204 02:53:31.560532 925 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:31.588830 909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e85e1dc6a5d4568ad365ba1d77398ff. Latest consensus state: current_term: 1 leader_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:31.589100 909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:31.592319 928 mvcc.cc:204] Tried to move back new op lower bound from 7250631112930734080 to 7250631112881799168. Current Snapshot: MvccSnapshot[applied={T|T < 7250631112930734080}]
I20260204 02:53:31.592960 873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e85e1dc6a5d4568ad365ba1d77398ff. Latest consensus state: current_term: 1 leader_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:31.593228 873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:31.602993 909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:31.603163 909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:31.603112 891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:31.603454 891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: This master's current role is: LEADER
I20260204 02:53:31.603681 891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:31.603837 891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: This master's current role is: LEADER
I20260204 02:53:31.605423 927 catalog_manager.cc:1357] Generated new cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:31.605510 927 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260204 02:53:31.622781 873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:31.623029 873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:31.635504 927 catalog_manager.cc:1380] Generated new certificate authority record
I20260204 02:53:31.638216 927 catalog_manager.cc:1514] Loading token signing keys...
I20260204 02:53:31.655864 927 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: Generated new TSK 0
I20260204 02:53:31.656791 927 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260204 02:53:31.657909 927 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260204 02:53:31.658681 927 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260204 02:53:31.690623 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.129:0
--local_ip_for_outbound_sockets=127.0.104.129
--webserver_interface=127.0.104.129
--webserver_port=0
--tserver_master_addrs=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20260204 02:53:32.011782 937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:53:32.012115 937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:53:32.012272 937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260204 02:53:32.021646 937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:53:32.021870 937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.129
I20260204 02:53:32.031280 937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.129
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:53:32.032872 937 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:53:32.034600 937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260204 02:53:32.043558 942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:32.043632 943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:32.044076 945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:53:32.044425 937 server_base.cc:1060] running on GCE node
I20260204 02:53:32.044888 937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:53:32.045706 937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:53:32.046914 937 hybrid_clock.cc:648] HybridClock initialized: now 1770173612046874 us; error 48 us; skew 500 ppm
I20260204 02:53:32.047206 937 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:53:32.049571 937 webserver.cc:492] Webserver started at http://127.0.104.129:46377/ using document root <none> and password file <none>
I20260204 02:53:32.050267 937 fs_manager.cc:362] Metadata directory not provided
I20260204 02:53:32.050390 937 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:53:32.050724 937 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260204 02:53:32.052965 937 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/data/instance:
uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
format_stamp: "Formatted at 2026-02-04 02:53:32 on dist-test-slave-xxhz"
I20260204 02:53:32.053699 937 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/wal/instance:
uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
format_stamp: "Formatted at 2026-02-04 02:53:32 on dist-test-slave-xxhz"
I20260204 02:53:32.058660 937 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260204 02:53:32.061681 951 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:32.062955 937 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260204 02:53:32.063107 937 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/wal
uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
format_stamp: "Formatted at 2026-02-04 02:53:32 on dist-test-slave-xxhz"
I20260204 02:53:32.063298 937 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:53:32.088590 937 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:53:32.089493 937 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:53:32.089761 937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260204 02:53:32.090622 937 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260204 02:53:32.092150 937 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260204 02:53:32.092262 937 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:32.092384 937 ts_tablet_manager.cc:616] Registered 0 tablets
I20260204 02:53:32.092463 937 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:32.125605 937 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.129:45767
I20260204 02:53:32.125654 1063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.129:45767 every 8 connection(s)
I20260204 02:53:32.126899 937 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/data/info.pb
I20260204 02:53:32.129683 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 937
I20260204 02:53:32.129930 418 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/wal/instance
I20260204 02:53:32.151757 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.130:0
--local_ip_for_outbound_sockets=127.0.104.130
--webserver_interface=127.0.104.130
--webserver_port=0
--tserver_master_addrs=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260204 02:53:32.155593 1074 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e85e1dc6a5d4568ad365ba1d77398ff)
I20260204 02:53:32.155827 1074 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:32.157920 840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78eb37be156448e8879b0df55832ab2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "202e4a0debb24c8aa89a4724d72fcf37" is_pre_election: true
I20260204 02:53:32.158980 1074 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
I20260204 02:53:32.159359 700 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78eb37be156448e8879b0df55832ab2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" is_pre_election: true
I20260204 02:53:32.160104 746 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:32.160472 1074 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:53:32.160936 1064 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:32.161334 1064 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:32.161451 1065 heartbeater.cc:344] Connected to a master server at 127.0.104.190:33671
I20260204 02:53:32.161659 1065 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:32.162457 1065 heartbeater.cc:507] Master 127.0.104.190:33671 requested a full tablet report, sending...
I20260204 02:53:32.162477 1064 heartbeater.cc:507] Master 127.0.104.188:38789 requested a full tablet report, sending...
I20260204 02:53:32.162740 1066 heartbeater.cc:344] Connected to a master server at 127.0.104.189:46765
I20260204 02:53:32.162870 1066 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:32.163836 1066 heartbeater.cc:507] Master 127.0.104.189:46765 requested a full tablet report, sending...
I20260204 02:53:32.164784 690 ts_manager.cc:194] Registered new tserver with Master: 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:32.164784 830 ts_manager.cc:194] Registered new tserver with Master: 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:32.165069 760 ts_manager.cc:194] Registered new tserver with Master: 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:32.166476 690 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.129:59731
I20260204 02:53:32.234877 887 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:32.235000 887 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: loading cluster ID for follower catalog manager: success
I20260204 02:53:32.238318 887 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: acquiring CA information for follower catalog manager: success
I20260204 02:53:32.240219 887 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260204 02:53:32.489398 1075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:53:32.489804 1075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:53:32.489950 1075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260204 02:53:32.493301 924 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:32.493418 924 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: loading cluster ID for follower catalog manager: success
I20260204 02:53:32.497481 924 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: acquiring CA information for follower catalog manager: success
I20260204 02:53:32.500205 924 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260204 02:53:32.501480 1075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:53:32.501681 1075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.130
I20260204 02:53:32.511268 1075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.130
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:53:32.512698 1075 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:53:32.514439 1075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260204 02:53:32.523963 1081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:32.524058 1082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:32.525336 1084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:53:32.526526 1075 server_base.cc:1060] running on GCE node
I20260204 02:53:32.527191 1075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:53:32.528316 1075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:53:32.529562 1075 hybrid_clock.cc:648] HybridClock initialized: now 1770173612529498 us; error 48 us; skew 500 ppm
I20260204 02:53:32.529907 1075 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:53:32.532987 1075 webserver.cc:492] Webserver started at http://127.0.104.130:40391/ using document root <none> and password file <none>
I20260204 02:53:32.533882 1075 fs_manager.cc:362] Metadata directory not provided
I20260204 02:53:32.533998 1075 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:53:32.534420 1075 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260204 02:53:32.537518 1075 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/data/instance:
uuid: "4dc2992e324049698330803c9ed2e3f8"
format_stamp: "Formatted at 2026-02-04 02:53:32 on dist-test-slave-xxhz"
I20260204 02:53:32.538496 1075 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/wal/instance:
uuid: "4dc2992e324049698330803c9ed2e3f8"
format_stamp: "Formatted at 2026-02-04 02:53:32 on dist-test-slave-xxhz"
I20260204 02:53:32.544896 1075 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260204 02:53:32.549393 1090 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:32.550861 1075 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260204 02:53:32.551016 1075 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/wal
uuid: "4dc2992e324049698330803c9ed2e3f8"
format_stamp: "Formatted at 2026-02-04 02:53:32 on dist-test-slave-xxhz"
I20260204 02:53:32.551234 1075 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:53:32.570477 1075 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:53:32.571388 1075 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:53:32.571687 1075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260204 02:53:32.572724 1075 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260204 02:53:32.574654 1075 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260204 02:53:32.574754 1075 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:32.574852 1075 ts_tablet_manager.cc:616] Registered 0 tablets
I20260204 02:53:32.574911 1075 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:32.617321 1075 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.130:44571
I20260204 02:53:32.617322 1202 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.130:44571 every 8 connection(s)
I20260204 02:53:32.619010 1075 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/data/info.pb
I20260204 02:53:32.628510 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 1075
I20260204 02:53:32.628697 418 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/wal/instance
I20260204 02:53:32.636471 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.190:33671
I20260204 02:53:32.636919 1204 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:32.636984 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.131:0
--local_ip_for_outbound_sockets=127.0.104.131
--webserver_interface=127.0.104.131
--webserver_port=0
--tserver_master_addrs=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260204 02:53:32.637989 1204 heartbeater.cc:507] Master 127.0.104.190:33671 requested a full tablet report, sending...
I20260204 02:53:32.639570 690 ts_manager.cc:194] Registered new tserver with Master: 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:32.640519 690 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.130:47609
I20260204 02:53:32.653479 1203 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:32.653919 1203 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:32.654304 1203 heartbeater.cc:507] Master 127.0.104.188:38789 requested a full tablet report, sending...
I20260204 02:53:32.655521 1211 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e85e1dc6a5d4568ad365ba1d77398ff)
I20260204 02:53:32.655709 1211 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:32.657642 1211 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
I20260204 02:53:32.658128 1205 heartbeater.cc:344] Connected to a master server at 127.0.104.189:46765
I20260204 02:53:32.658439 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202e4a0debb24c8aa89a4724d72fcf37" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78eb37be156448e8879b0df55832ab2d" is_pre_election: true
I20260204 02:53:32.658839 700 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202e4a0debb24c8aa89a4724d72fcf37" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" is_pre_election: true
I20260204 02:53:32.659467 1205 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:32.659554 816 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:32.659865 1205 heartbeater.cc:507] Master 127.0.104.189:46765 requested a full tablet report, sending...
I20260204 02:53:32.660396 830 ts_manager.cc:194] Registered new tserver with Master: 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:32.661058 1211 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:53:32.662477 760 ts_manager.cc:194] Registered new tserver with Master: 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:32.669094 1065 heartbeater.cc:499] Master 127.0.104.190:33671 was elected leader, sending a full tablet report...
I20260204 02:53:32.731262 1212 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e85e1dc6a5d4568ad365ba1d77398ff)
I20260204 02:53:32.731640 1212 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:32.732745 1212 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
I20260204 02:53:32.733055 840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78eb37be156448e8879b0df55832ab2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "202e4a0debb24c8aa89a4724d72fcf37" is_pre_election: true
I20260204 02:53:32.733417 700 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78eb37be156448e8879b0df55832ab2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" is_pre_election: true
I20260204 02:53:32.733940 746 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:32.734228 1212 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260204 02:53:33.003108 1210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:53:33.003448 1210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:53:33.003639 1210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260204 02:53:33.011829 1210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:53:33.012039 1210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.131
I20260204 02:53:33.021811 1210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.131
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:53:33.023524 1210 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:53:33.025319 1210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260204 02:53:33.033880 1218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:33.033794 1217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:33.034744 1220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:53:33.035049 1210 server_base.cc:1060] running on GCE node
I20260204 02:53:33.035629 1210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:53:33.036993 1210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:53:33.038210 1210 hybrid_clock.cc:648] HybridClock initialized: now 1770173613038153 us; error 48 us; skew 500 ppm
I20260204 02:53:33.038543 1210 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:53:33.040489 1210 webserver.cc:492] Webserver started at http://127.0.104.131:40315/ using document root <none> and password file <none>
I20260204 02:53:33.041160 1210 fs_manager.cc:362] Metadata directory not provided
I20260204 02:53:33.041285 1210 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:53:33.041626 1210 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260204 02:53:33.043864 1210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/data/instance:
uuid: "e2efdafb1246484a98dc5408be6746b5"
format_stamp: "Formatted at 2026-02-04 02:53:33 on dist-test-slave-xxhz"
I20260204 02:53:33.044551 1210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal/instance:
uuid: "e2efdafb1246484a98dc5408be6746b5"
format_stamp: "Formatted at 2026-02-04 02:53:33 on dist-test-slave-xxhz"
I20260204 02:53:33.049646 1210 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260204 02:53:33.052639 1226 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:33.053946 1210 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260204 02:53:33.054162 1210 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal
uuid: "e2efdafb1246484a98dc5408be6746b5"
format_stamp: "Formatted at 2026-02-04 02:53:33 on dist-test-slave-xxhz"
I20260204 02:53:33.054375 1210 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:53:33.068877 1210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:53:33.069648 1210 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:53:33.069938 1210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260204 02:53:33.070784 1210 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260204 02:53:33.072568 1210 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260204 02:53:33.072701 1210 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:33.072837 1210 ts_tablet_manager.cc:616] Registered 0 tablets
I20260204 02:53:33.072917 1210 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:33.108225 1210 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.131:34247
I20260204 02:53:33.108260 1338 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.131:34247 every 8 connection(s)
I20260204 02:53:33.109496 1210 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/data/info.pb
I20260204 02:53:33.116190 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 1210
I20260204 02:53:33.116459 418 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal/instance
I20260204 02:53:33.136539 1340 heartbeater.cc:344] Connected to a master server at 127.0.104.190:33671
I20260204 02:53:33.136541 1339 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:33.137321 1341 heartbeater.cc:344] Connected to a master server at 127.0.104.189:46765
I20260204 02:53:33.137362 1339 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:33.137494 1341 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:33.138362 1341 heartbeater.cc:507] Master 127.0.104.189:46765 requested a full tablet report, sending...
I20260204 02:53:33.138391 1339 heartbeater.cc:507] Master 127.0.104.188:38789 requested a full tablet report, sending...
I20260204 02:53:33.139219 1340 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:33.139866 1340 heartbeater.cc:507] Master 127.0.104.190:33671 requested a full tablet report, sending...
I20260204 02:53:33.139953 830 ts_manager.cc:194] Registered new tserver with Master: e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:33.140062 760 ts_manager.cc:194] Registered new tserver with Master: e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:33.140678 690 ts_manager.cc:194] Registered new tserver with Master: e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:33.141592 690 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.131:48117
I20260204 02:53:33.143517 418 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260204 02:53:33.146437 1204 heartbeater.cc:499] Master 127.0.104.190:33671 was elected leader, sending a full tablet report...
I20260204 02:53:33.173499 689 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59762:
name: "default.Table_0312705ce5e140d4bcf88a92cab2bb7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.173842 688 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59762:
name: "default.Table_69c350f50a63410c9da5a5e7b0862cf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.174059 1358 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e85e1dc6a5d4568ad365ba1d77398ff)
I20260204 02:53:33.174216 1358 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:33.174904 1358 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
I20260204 02:53:33.175329 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202e4a0debb24c8aa89a4724d72fcf37" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78eb37be156448e8879b0df55832ab2d" is_pre_election: true
I20260204 02:53:33.177042 687 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59762:
name: "default.Table_cb3f03c12e314422b462640be08ec293"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.179315 686 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59762:
name: "default.Table_3d3a816d407e429ea458b787f879ede0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.179406 685 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59762:
name: "default.Table_a16e01078d0342e7abbb4ec634032416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.180015 684 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59762:
name: "default.Table_3a403dbfa3044e55abc223b3acd75b6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.180774 683 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59762:
name: "default.Table_79ae542654da4da5aba4d8c38d6abae7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.181552 682 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59762:
name: "default.Table_db7b7860261b41c692b36ca95b2d89e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.183225 690 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59762:
name: "default.Table_842bb3fe3b2944498784e63317dd1907"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.188019 700 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202e4a0debb24c8aa89a4724d72fcf37" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" is_pre_election: true
I20260204 02:53:33.188731 816 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:33.189059 1358 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:53:33.191885 681 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59762:
name: "default.Table_f734190e49874f9fb2c8dee62aa9e70b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260204 02:53:33.192972 689 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0312705ce5e140d4bcf88a92cab2bb7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260204 02:53:33.208154 418 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task6hEvLx/build/asan/bin/kudu pid 659
I20260204 02:53:33.209498 720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260204 02:53:33.275805 1387 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e85e1dc6a5d4568ad365ba1d77398ff)
I20260204 02:53:33.276029 1387 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:33.276778 1387 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
I20260204 02:53:33.277428 840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78eb37be156448e8879b0df55832ab2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "202e4a0debb24c8aa89a4724d72fcf37" is_pre_election: true
I20260204 02:53:33.277643 840 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78eb37be156448e8879b0df55832ab2d in term 1.
I20260204 02:53:33.278095 744 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d; no voters:
I20260204 02:53:33.278375 1387 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260204 02:53:33.278476 1387 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3e85e1dc6a5d4568ad365ba1d77398ff)
I20260204 02:53:33.278561 1387 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 1 FOLLOWER]: Advancing to term 2
I20260204 02:53:33.280329 1387 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:33.280938 1387 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 2 election: Requested vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
I20260204 02:53:33.281497 840 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78eb37be156448e8879b0df55832ab2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "202e4a0debb24c8aa89a4724d72fcf37"
I20260204 02:53:33.281657 840 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 1 FOLLOWER]: Advancing to term 2
I20260204 02:53:33.283934 840 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78eb37be156448e8879b0df55832ab2d in term 2.
I20260204 02:53:33.284317 744 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d; no voters:
I20260204 02:53:33.284572 1387 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 2 FOLLOWER]: Leader election won for term 2
I20260204 02:53:33.284996 1387 raft_consensus.cc:697] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 2 LEADER]: Becoming Leader. State: Replica: 78eb37be156448e8879b0df55832ab2d, State: Running, Role: LEADER
I20260204 02:53:33.285449 1387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:33.287063 1388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78eb37be156448e8879b0df55832ab2d. Latest consensus state: current_term: 2 leader_uuid: "78eb37be156448e8879b0df55832ab2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:33.287281 1388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: LEADER
I20260204 02:53:33.288038 1390 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260204 02:53:33.290410 1390 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260204 02:53:33.291102 1390 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:33.291188 1390 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260204 02:53:33.292162 1390 catalog_manager.cc:1514] Loading token signing keys...
I20260204 02:53:33.292860 1390 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Loaded TSK: 0
I20260204 02:53:33.293666 1390 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260204 02:53:33.294510 1390 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260204 02:53:33.295881 1390 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260204 02:53:33.380569 769 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 2 LEADER]: Rejecting Update request from peer 3e85e1dc6a5d4568ad365ba1d77398ff for earlier term 1. Current term is 2. Ops: []
I20260204 02:53:33.380812 840 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 2 FOLLOWER]: Rejecting Update request from peer 3e85e1dc6a5d4568ad365ba1d77398ff for earlier term 1. Current term is 2. Ops: []
I20260204 02:53:33.387784 1391 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260204 02:53:33.388067 418 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6hEvLx/build/asan/bin/kudu with pid 659
W20260204 02:53:33.414175 1348 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.190:33671)
W20260204 02:53:33.415230 746 connection.cc:537] client connection to 127.0.104.190:33671 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:53:33.420411 1352 negotiation.cc:337] Failed RPC negotiation. Trace:
0204 02:53:33.415640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.190:33671 (local address 127.0.0.1:59774)
0204 02:53:33.416071 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0204 02:53:33.416096 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0204 02:53:33.416321 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0204 02:53:33.418665 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":119}
W20260204 02:53:33.422173 746 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:33.422501 746 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:53:33.429136 751 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_a16e01078d0342e7abbb4ec634032416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.427331 754 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_842bb3fe3b2944498784e63317dd1907"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.426403 757 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_3a403dbfa3044e55abc223b3acd75b6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.423380 760 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_f734190e49874f9fb2c8dee62aa9e70b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.425540 759 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_db7b7860261b41c692b36ca95b2d89e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.426690 755 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_0312705ce5e140d4bcf88a92cab2bb7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.427783 752 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_3d3a816d407e429ea458b787f879ede0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.427258 753 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_cb3f03c12e314422b462640be08ec293"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.425598 758 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_79ae542654da4da5aba4d8c38d6abae7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:33.426326 756 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_69c350f50a63410c9da5a5e7b0862cf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260204 02:53:33.438544 757 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3a403dbfa3044e55abc223b3acd75b6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260204 02:53:33.643922 1340 heartbeater.cc:499] Master 127.0.104.190:33671 was elected leader, sending a full tablet report...
W20260204 02:53:33.646313 1340 heartbeater.cc:646] Failed to heartbeat to 127.0.104.190:33671 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:33.649382 1204 heartbeater.cc:646] Failed to heartbeat to 127.0.104.190:33671 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:33.678067 1065 heartbeater.cc:646] Failed to heartbeat to 127.0.104.190:33671 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:53:33.736394 757 catalog_manager.cc:2174] created HMS catalog entry for table [id=333921075666492fb4e06fa35e04487b]
W20260204 02:53:33.738857 754 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_842bb3fe3b2944498784e63317dd1907 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260204 02:53:33.739940 760 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f734190e49874f9fb2c8dee62aa9e70b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260204 02:53:33.740415 840 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 2 FOLLOWER]: Refusing update from remote peer 78eb37be156448e8879b0df55832ab2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20260204 02:53:33.740849 759 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_db7b7860261b41c692b36ca95b2d89e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260204 02:53:33.741039 1387 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260204 02:53:33.741670 755 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0312705ce5e140d4bcf88a92cab2bb7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260204 02:53:33.742537 752 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3d3a816d407e429ea458b787f879ede0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260204 02:53:33.743580 753 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cb3f03c12e314422b462640be08ec293 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260204 02:53:33.744688 758 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_79ae542654da4da5aba4d8c38d6abae7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260204 02:53:33.745772 756 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_69c350f50a63410c9da5a5e7b0862cf6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260204 02:53:33.747802 751 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a16e01078d0342e7abbb4ec634032416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260204 02:53:33.750198 746 consensus_peers.cc:597] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d -> Peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Couldn't send request to peer 3e85e1dc6a5d4568ad365ba1d77398ff. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260204 02:53:33.759270 1398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78eb37be156448e8879b0df55832ab2d. Latest consensus state: current_term: 2 leader_uuid: "78eb37be156448e8879b0df55832ab2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:33.759548 1398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:33.765157 1397 mvcc.cc:204] Tried to move back new op lower bound from 7250631121874235392 to 7250631120020959232. Current Snapshot: MvccSnapshot[applied={T|T < 7250631121874235392}]
I20260204 02:53:33.770150 1387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "78eb37be156448e8879b0df55832ab2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:33.770382 1387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: LEADER
I20260204 02:53:33.771586 1398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "78eb37be156448e8879b0df55832ab2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:33.771785 1398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:33.779970 757 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38314:
table { table_name: "default.Table_3a403dbfa3044e55abc223b3acd75b6b" } new_table_name: "default.Table_7021e56527ac45df985d01556abaeeb8" modify_external_catalogs: true
I20260204 02:53:33.783070 754 catalog_manager.cc:2174] created HMS catalog entry for table [id=22158e83a9534758a87a9a1f10eaba76]
I20260204 02:53:33.809394 1274 tablet_service.cc:1505] Processing CreateTablet for tablet 81d3882b863f4080a184bb084fcbff94 (DEFAULT_TABLE table=default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:33.810631 1274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d3882b863f4080a184bb084fcbff94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:33.818646 760 catalog_manager.cc:2174] created HMS catalog entry for table [id=7a55ab36a42842a6a7b1f572501cd1f2]
I20260204 02:53:33.825544 1138 tablet_service.cc:1505] Processing CreateTablet for tablet 81d3882b863f4080a184bb084fcbff94 (DEFAULT_TABLE table=default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:33.827104 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d3882b863f4080a184bb084fcbff94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:33.837543 754 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38314:
table { table_name: "default.Table_842bb3fe3b2944498784e63317dd1907" } new_table_name: "default.Table_fb4d5c02e3f443afb1f2f219731403b1" modify_external_catalogs: true
I20260204 02:53:33.843902 760 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38314:
table { table_name: "default.Table_f734190e49874f9fb2c8dee62aa9e70b" } new_table_name: "default.Table_f9d078d036094cf486755f573c75e8b6" modify_external_catalogs: true
I20260204 02:53:33.850142 1273 tablet_service.cc:1505] Processing CreateTablet for tablet 64e25eca7e004205b329cb37a6e9471f (DEFAULT_TABLE table=default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:33.850649 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64e25eca7e004205b329cb37a6e9471f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:33.853850 1137 tablet_service.cc:1505] Processing CreateTablet for tablet 64e25eca7e004205b329cb37a6e9471f (DEFAULT_TABLE table=default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:33.854405 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64e25eca7e004205b329cb37a6e9471f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:33.865335 1409 tablet_bootstrap.cc:492] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:33.869866 1409 tablet_bootstrap.cc:654] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:33.871014 999 tablet_service.cc:1505] Processing CreateTablet for tablet 81d3882b863f4080a184bb084fcbff94 (DEFAULT_TABLE table=default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:33.873036 999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d3882b863f4080a184bb084fcbff94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:33.875279 1409 log.cc:826] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8: Log is configured to *not* fsync() on all Append() calls
I20260204 02:53:33.878918 1273 tablet_service.cc:1505] Processing CreateTablet for tablet ef445dad9acd4c69a9fb5d03dfe1df5f (DEFAULT_TABLE table=default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:33.879352 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef445dad9acd4c69a9fb5d03dfe1df5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:33.880134 1137 tablet_service.cc:1505] Processing CreateTablet for tablet ef445dad9acd4c69a9fb5d03dfe1df5f (DEFAULT_TABLE table=default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:33.880539 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef445dad9acd4c69a9fb5d03dfe1df5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:33.886050 759 catalog_manager.cc:2174] created HMS catalog entry for table [id=8dded49817604d999f92e9157655e2f2]
I20260204 02:53:33.889701 1409 tablet_bootstrap.cc:492] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:33.889977 1409 ts_tablet_manager.cc:1403] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.007s
I20260204 02:53:33.893358 759 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38314:
table { table_name: "default.Table_db7b7860261b41c692b36ca95b2d89e2" } new_table_name: "default.Table_667ea74f761c447c8538d777329b0b9f" modify_external_catalogs: true
I20260204 02:53:33.902232 1409 raft_consensus.cc:359] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:33.902616 1409 raft_consensus.cc:385] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:33.902719 1409 raft_consensus.cc:740] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:33.907694 998 tablet_service.cc:1505] Processing CreateTablet for tablet 64e25eca7e004205b329cb37a6e9471f (DEFAULT_TABLE table=default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:33.907799 997 tablet_service.cc:1505] Processing CreateTablet for tablet ef445dad9acd4c69a9fb5d03dfe1df5f (DEFAULT_TABLE table=default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:33.908247 998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64e25eca7e004205b329cb37a6e9471f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:33.908767 1408 tablet_bootstrap.cc:492] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:33.911645 996 tablet_service.cc:1505] Processing CreateTablet for tablet f66e2750d8c343c0bfa567fd36bc4da0 (DEFAULT_TABLE table=default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:33.911144 1273 tablet_service.cc:1505] Processing CreateTablet for tablet f66e2750d8c343c0bfa567fd36bc4da0 (DEFAULT_TABLE table=default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:33.911774 1137 tablet_service.cc:1505] Processing CreateTablet for tablet f66e2750d8c343c0bfa567fd36bc4da0 (DEFAULT_TABLE table=default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:33.912127 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f66e2750d8c343c0bfa567fd36bc4da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:33.912145 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f66e2750d8c343c0bfa567fd36bc4da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:33.915846 1408 tablet_bootstrap.cc:654] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:33.917887 1408 log.cc:826] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5: Log is configured to *not* fsync() on all Append() calls
I20260204 02:53:33.918668 996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f66e2750d8c343c0bfa567fd36bc4da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:33.918960 1409 consensus_queue.cc:260] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:33.923851 997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef445dad9acd4c69a9fb5d03dfe1df5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:33.944762 1413 tablet_bootstrap.cc:492] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:33.945029 1409 ts_tablet_manager.cc:1434] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.055s user 0.020s sys 0.009s
I20260204 02:53:33.947957 1413 tablet_bootstrap.cc:654] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:33.949200 1413 log.cc:826] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Log is configured to *not* fsync() on all Append() calls
I20260204 02:53:33.953974 1205 heartbeater.cc:499] Master 127.0.104.189:46765 was elected leader, sending a full tablet report...
I20260204 02:53:33.956480 1409 tablet_bootstrap.cc:492] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:33.958364 1413 tablet_bootstrap.cc:492] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:33.958379 1409 tablet_bootstrap.cc:654] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:33.958675 1413 ts_tablet_manager.cc:1403] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260204 02:53:33.961576 1408 tablet_bootstrap.cc:492] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:33.961856 1408 ts_tablet_manager.cc:1403] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.022s
I20260204 02:53:33.965385 1413 raft_consensus.cc:359] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:33.965747 1413 raft_consensus.cc:385] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:33.965834 1413 raft_consensus.cc:740] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:33.966513 1413 consensus_queue.cc:260] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:33.968823 1408 raft_consensus.cc:359] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:33.969353 1408 raft_consensus.cc:385] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:33.969540 1408 raft_consensus.cc:740] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:33.970381 1409 tablet_bootstrap.cc:492] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:33.970530 1409 ts_tablet_manager.cc:1403] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20260204 02:53:33.970573 1408 consensus_queue.cc:260] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:33.971609 1409 raft_consensus.cc:359] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:33.971843 1409 raft_consensus.cc:385] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:33.971959 1409 raft_consensus.cc:740] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:33.972221 1409 consensus_queue.cc:260] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:33.972747 1409 ts_tablet_manager.cc:1434] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260204 02:53:33.979110 1409 tablet_bootstrap.cc:492] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:33.981344 1409 tablet_bootstrap.cc:654] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:33.987139 1409 tablet_bootstrap.cc:492] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:33.987360 1409 ts_tablet_manager.cc:1403] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260204 02:53:33.988353 1409 raft_consensus.cc:359] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:33.988600 1409 raft_consensus.cc:385] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:33.988694 1409 raft_consensus.cc:740] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:33.988991 1409 consensus_queue.cc:260] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:33.989499 1409 ts_tablet_manager.cc:1434] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260204 02:53:33.989499 1066 heartbeater.cc:499] Master 127.0.104.189:46765 was elected leader, sending a full tablet report...
I20260204 02:53:33.990829 1413 ts_tablet_manager.cc:1434] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.032s user 0.020s sys 0.004s
I20260204 02:53:33.991271 1413 tablet_bootstrap.cc:492] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:33.993445 1409 tablet_bootstrap.cc:492] T f66e2750d8c343c0bfa567fd36bc4da0 P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:33.995334 1409 tablet_bootstrap.cc:654] T f66e2750d8c343c0bfa567fd36bc4da0 P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:33.997092 1413 tablet_bootstrap.cc:654] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:33.997920 1341 heartbeater.cc:499] Master 127.0.104.189:46765 was elected leader, sending a full tablet report...
I20260204 02:53:33.998817 1408 ts_tablet_manager.cc:1434] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.037s user 0.021s sys 0.001s
I20260204 02:53:33.999336 1408 tablet_bootstrap.cc:492] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:34.001516 1408 tablet_bootstrap.cc:654] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.003764 1409 tablet_bootstrap.cc:492] T f66e2750d8c343c0bfa567fd36bc4da0 P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:34.003949 1409 ts_tablet_manager.cc:1403] T f66e2750d8c343c0bfa567fd36bc4da0 P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260204 02:53:34.004742 1409 raft_consensus.cc:359] T f66e2750d8c343c0bfa567fd36bc4da0 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.004999 1409 raft_consensus.cc:385] T f66e2750d8c343c0bfa567fd36bc4da0 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.005134 1409 raft_consensus.cc:740] T f66e2750d8c343c0bfa567fd36bc4da0 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.005654 1409 consensus_queue.cc:260] T f66e2750d8c343c0bfa567fd36bc4da0 P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.008925 1409 ts_tablet_manager.cc:1434] T f66e2750d8c343c0bfa567fd36bc4da0 P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260204 02:53:34.012601 1413 tablet_bootstrap.cc:492] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:34.012759 1413 ts_tablet_manager.cc:1403] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.007s
I20260204 02:53:34.013018 1408 tablet_bootstrap.cc:492] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:34.013151 1408 ts_tablet_manager.cc:1403] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260204 02:53:34.013753 1413 raft_consensus.cc:359] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.013934 1413 raft_consensus.cc:385] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.013947 1408 raft_consensus.cc:359] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.014037 1413 raft_consensus.cc:740] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.014117 1408 raft_consensus.cc:385] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.014205 1408 raft_consensus.cc:740] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.014302 1413 consensus_queue.cc:260] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.014436 1408 consensus_queue.cc:260] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.014901 1408 ts_tablet_manager.cc:1434] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260204 02:53:34.015264 1408 tablet_bootstrap.cc:492] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:34.016417 1413 ts_tablet_manager.cc:1434] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260204 02:53:34.017069 1408 tablet_bootstrap.cc:654] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.017601 1413 tablet_bootstrap.cc:492] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:34.019299 1413 tablet_bootstrap.cc:654] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.019423 1408 tablet_bootstrap.cc:492] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:34.019589 1408 ts_tablet_manager.cc:1403] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260204 02:53:34.020334 1408 raft_consensus.cc:359] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.020542 1408 raft_consensus.cc:385] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.020641 1408 raft_consensus.cc:740] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.020977 1408 consensus_queue.cc:260] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.021562 1408 ts_tablet_manager.cc:1434] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260204 02:53:34.021896 1408 tablet_bootstrap.cc:492] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:34.023965 1408 tablet_bootstrap.cc:654] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.027446 1413 tablet_bootstrap.cc:492] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:34.027750 1413 ts_tablet_manager.cc:1403] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260204 02:53:34.028740 1413 raft_consensus.cc:359] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.029086 1413 raft_consensus.cc:385] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.029233 1413 raft_consensus.cc:740] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.029665 1413 consensus_queue.cc:260] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.032582 1408 tablet_bootstrap.cc:492] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:34.032768 1408 ts_tablet_manager.cc:1403] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260204 02:53:34.032989 1413 ts_tablet_manager.cc:1434] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260204 02:53:34.034106 1413 tablet_bootstrap.cc:492] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:34.034170 1408 raft_consensus.cc:359] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.034384 1408 raft_consensus.cc:385] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.034514 1408 raft_consensus.cc:740] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.034852 1408 consensus_queue.cc:260] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.035426 1408 ts_tablet_manager.cc:1434] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260204 02:53:34.035892 1413 tablet_bootstrap.cc:654] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.038211 1413 tablet_bootstrap.cc:492] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:34.038379 1413 ts_tablet_manager.cc:1403] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260204 02:53:34.039132 1413 raft_consensus.cc:359] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.039306 1413 raft_consensus.cc:385] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.039413 1413 raft_consensus.cc:740] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.039676 1413 consensus_queue.cc:260] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.040637 1413 ts_tablet_manager.cc:1434] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260204 02:53:34.042348 1412 raft_consensus.cc:493] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:34.042577 1412 raft_consensus.cc:515] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.044505 1412 leader_election.cc:290] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:34.052572 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef445dad9acd4c69a9fb5d03dfe1df5f" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:34.053110 1019 raft_consensus.cc:2468] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 0.
I20260204 02:53:34.055027 1091 leader_election.cc:304] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:34.055562 1412 raft_consensus.cc:2804] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:34.055711 1412 raft_consensus.cc:493] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:34.055805 1412 raft_consensus.cc:3060] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.059129 1412 raft_consensus.cc:515] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.060494 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef445dad9acd4c69a9fb5d03dfe1df5f" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
I20260204 02:53:34.060737 1019 raft_consensus.cc:3060] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.064407 1019 raft_consensus.cc:2468] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
I20260204 02:53:34.064837 1091 leader_election.cc:304] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:34.065723 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef445dad9acd4c69a9fb5d03dfe1df5f" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5" is_pre_election: true
I20260204 02:53:34.066150 1294 raft_consensus.cc:2468] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 0.
I20260204 02:53:34.069193 1412 leader_election.cc:290] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 election: Requested vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:34.069622 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef445dad9acd4c69a9fb5d03dfe1df5f" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5"
I20260204 02:53:34.069757 1412 raft_consensus.cc:2804] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:34.069825 1294 raft_consensus.cc:3060] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.069878 1412 raft_consensus.cc:697] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Running, Role: LEADER
I20260204 02:53:34.070372 1412 consensus_queue.cc:237] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.073151 1294 raft_consensus.cc:2468] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
02:53:34.083 [HMS - ERROR - pool-6-thread-3] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_0312705ce5e140d4bcf88a92cab2bb7b already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20260204 02:53:34.085253 755 catalog_manager.cc:2170] Already present: failed to create HMS catalog entry for table [id=78f3ce9a0522476592bdcbd745dcbe01]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_0312705ce5e140d4bcf88a92cab2bb7b already exists)
I20260204 02:53:34.090279 1427 raft_consensus.cc:493] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:34.090567 1427 raft_consensus.cc:515] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:34.091938 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81d3882b863f4080a184bb084fcbff94" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5" is_pre_election: true
I20260204 02:53:34.091936 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81d3882b863f4080a184bb084fcbff94" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:34.092146 1294 raft_consensus.cc:2468] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 0.
I20260204 02:53:34.092146 1019 raft_consensus.cc:2468] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 0.
I20260204 02:53:34.092574 1091 leader_election.cc:304] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:34.092604 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.190:33671
--webserver_interface=127.0.104.190
--webserver_port=46841
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260204 02:53:34.092710 1427 leader_election.cc:290] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:34.092855 1427 raft_consensus.cc:2804] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:34.092936 1427 raft_consensus.cc:493] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:34.093003 1427 raft_consensus.cc:3060] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.095580 1427 raft_consensus.cc:515] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:34.097460 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81d3882b863f4080a184bb084fcbff94" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5"
I20260204 02:53:34.097679 1294 raft_consensus.cc:3060] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.100085 1294 raft_consensus.cc:2468] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
I20260204 02:53:34.100801 1093 leader_election.cc:304] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:34.101135 1412 raft_consensus.cc:2804] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:34.103008 1427 leader_election.cc:290] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 election: Requested vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:34.103442 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81d3882b863f4080a184bb084fcbff94" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
I20260204 02:53:34.103654 1019 raft_consensus.cc:3060] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.105489 755 catalog_manager.cc:5654] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130). New cstate: current_term: 1 leader_uuid: "4dc2992e324049698330803c9ed2e3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } health_report { overall_health: UNKNOWN } } }
I20260204 02:53:34.105973 1019 raft_consensus.cc:2468] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
I20260204 02:53:34.107137 1412 raft_consensus.cc:697] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Running, Role: LEADER
I20260204 02:53:34.107574 1412 consensus_queue.cc:237] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
W20260204 02:53:34.114331 1342 tablet.cc:2378] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260204 02:53:34.125388 1416 raft_consensus.cc:493] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:34.125557 1416 raft_consensus.cc:515] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.127947 1416 leader_election.cc:290] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:53:34.132726 1067 tablet.cc:2378] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260204 02:53:34.138130 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f66e2750d8c343c0bfa567fd36bc4da0" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8" is_pre_election: true
I20260204 02:53:34.141613 755 master_service.cc:954] ReplaceTablet: received request to replace tablet ef445dad9acd4c69a9fb5d03dfe1df5f from {username='slave'} at 127.0.0.1:38314
I20260204 02:53:34.147148 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f66e2750d8c343c0bfa567fd36bc4da0" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5" is_pre_election: true
I20260204 02:53:34.147341 1294 raft_consensus.cc:2468] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 0.
I20260204 02:53:34.147928 954 leader_election.cc:304] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:34.148295 1416 raft_consensus.cc:2804] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:34.148420 1416 raft_consensus.cc:493] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:34.148393 1158 raft_consensus.cc:2468] T f66e2750d8c343c0bfa567fd36bc4da0 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 0.
I20260204 02:53:34.148501 1416 raft_consensus.cc:3060] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
W20260204 02:53:34.148882 1208 tablet.cc:2378] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260204 02:53:34.151113 1416 raft_consensus.cc:515] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.151705 1416 leader_election.cc:290] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 election: Requested vote from peers 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:34.152304 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f66e2750d8c343c0bfa567fd36bc4da0" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8"
I20260204 02:53:34.152482 1158 raft_consensus.cc:3060] T f66e2750d8c343c0bfa567fd36bc4da0 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.152671 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f66e2750d8c343c0bfa567fd36bc4da0" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5"
I20260204 02:53:34.152860 1294 raft_consensus.cc:3060] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.155297 1294 raft_consensus.cc:2468] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 1.
I20260204 02:53:34.155297 1158 raft_consensus.cc:2468] T f66e2750d8c343c0bfa567fd36bc4da0 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 1.
I20260204 02:53:34.155927 952 leader_election.cc:304] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:34.156167 755 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Sending DeleteTablet for 3 replicas of tablet ef445dad9acd4c69a9fb5d03dfe1df5f
I20260204 02:53:34.156553 1416 raft_consensus.cc:2804] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:34.157321 999 tablet_service.cc:1552] Processing DeleteTablet for tablet ef445dad9acd4c69a9fb5d03dfe1df5f with delete_type TABLET_DATA_DELETED (replaced by tablet b8094eec37cd40b9b1f4931f24d824a8) from {username='slave'} at 127.0.0.1:45802
I20260204 02:53:34.157958 1137 tablet_service.cc:1552] Processing DeleteTablet for tablet ef445dad9acd4c69a9fb5d03dfe1df5f with delete_type TABLET_DATA_DELETED (replaced by tablet b8094eec37cd40b9b1f4931f24d824a8) from {username='slave'} at 127.0.0.1:56478
I20260204 02:53:34.158318 755 catalog_manager.cc:6724] ReplaceTablet: tablet ef445dad9acd4c69a9fb5d03dfe1df5f deleted and replaced by tablet b8094eec37cd40b9b1f4931f24d824a8
I20260204 02:53:34.158371 1274 tablet_service.cc:1552] Processing DeleteTablet for tablet ef445dad9acd4c69a9fb5d03dfe1df5f with delete_type TABLET_DATA_DELETED (replaced by tablet b8094eec37cd40b9b1f4931f24d824a8) from {username='slave'} at 127.0.0.1:42962
I20260204 02:53:34.169677 1138 tablet_service.cc:1505] Processing CreateTablet for tablet b8094eec37cd40b9b1f4931f24d824a8 (DEFAULT_TABLE table=default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.170228 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8094eec37cd40b9b1f4931f24d824a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.174191 998 tablet_service.cc:1505] Processing CreateTablet for tablet b8094eec37cd40b9b1f4931f24d824a8 (DEFAULT_TABLE table=default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.174628 998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8094eec37cd40b9b1f4931f24d824a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.177107 1435 tablet_replica.cc:333] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8: stopping tablet replica
I20260204 02:53:34.177572 1435 raft_consensus.cc:2243] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [term 1 LEADER]: Raft consensus shutting down.
I20260204 02:53:34.178185 1435 pending_rounds.cc:70] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8: Trying to abort 1 pending ops.
I20260204 02:53:34.178354 1435 pending_rounds.cc:77] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7250631123245035520 op_type: NO_OP noop_request { }
I20260204 02:53:34.178560 1435 raft_consensus.cc:2889] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260204 02:53:34.178670 1435 raft_consensus.cc:2272] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:34.171442 1436 tablet_replica.cc:333] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5: stopping tablet replica
I20260204 02:53:34.182802 1274 tablet_service.cc:1505] Processing CreateTablet for tablet b8094eec37cd40b9b1f4931f24d824a8 (DEFAULT_TABLE table=default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.183346 1274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8094eec37cd40b9b1f4931f24d824a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.189060 1413 tablet_bootstrap.cc:492] T b8094eec37cd40b9b1f4931f24d824a8 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:34.190608 1413 tablet_bootstrap.cc:654] T b8094eec37cd40b9b1f4931f24d824a8 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.192072 1434 tablet_replica.cc:333] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: stopping tablet replica
I20260204 02:53:34.192598 1434 raft_consensus.cc:2243] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260204 02:53:34.193081 1434 raft_consensus.cc:2272] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:34.193591 1416 raft_consensus.cc:697] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Running, Role: LEADER
I20260204 02:53:34.194072 1416 consensus_queue.cc:237] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.194355 1409 tablet_bootstrap.cc:492] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:34.195942 1409 tablet_bootstrap.cc:654] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.182931 1436 raft_consensus.cc:2243] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260204 02:53:34.208504 1436 raft_consensus.cc:2272] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:34.209285 755 catalog_manager.cc:5654] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130). New cstate: current_term: 1 leader_uuid: "4dc2992e324049698330803c9ed2e3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } health_report { overall_health: UNKNOWN } } }
I20260204 02:53:34.210263 1436 ts_tablet_manager.cc:1916] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:34.213433 1408 tablet_bootstrap.cc:492] T b8094eec37cd40b9b1f4931f24d824a8 P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:34.214687 1409 tablet_bootstrap.cc:492] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:34.215008 1409 ts_tablet_manager.cc:1403] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.006s
I20260204 02:53:34.215242 1408 tablet_bootstrap.cc:654] T b8094eec37cd40b9b1f4931f24d824a8 P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.215958 1409 raft_consensus.cc:359] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.216322 1409 raft_consensus.cc:385] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.216449 1409 raft_consensus.cc:740] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.216500 1436 ts_tablet_manager.cc:1929] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260204 02:53:34.216643 1436 log.cc:1199] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal/wals/ef445dad9acd4c69a9fb5d03dfe1df5f
I20260204 02:53:34.216885 1409 consensus_queue.cc:260] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.217446 1409 ts_tablet_manager.cc:1434] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260204 02:53:34.218936 1434 ts_tablet_manager.cc:1916] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:34.217113 1436 ts_tablet_manager.cc:1950] T ef445dad9acd4c69a9fb5d03dfe1df5f P e2efdafb1246484a98dc5408be6746b5: Deleting consensus metadata
I20260204 02:53:34.223591 752 catalog_manager.cc:2174] created HMS catalog entry for table [id=d8f5b0e4ab40417ba4f4b805c6c6941f]
I20260204 02:53:34.224004 1413 tablet_bootstrap.cc:492] T b8094eec37cd40b9b1f4931f24d824a8 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:34.224154 1413 ts_tablet_manager.cc:1403] T b8094eec37cd40b9b1f4931f24d824a8 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.003s
I20260204 02:53:34.225317 1413 raft_consensus.cc:359] T b8094eec37cd40b9b1f4931f24d824a8 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.225523 1413 raft_consensus.cc:385] T b8094eec37cd40b9b1f4931f24d824a8 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.225603 1413 raft_consensus.cc:740] T b8094eec37cd40b9b1f4931f24d824a8 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.226922 1413 consensus_queue.cc:260] T b8094eec37cd40b9b1f4931f24d824a8 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.228353 746 catalog_manager.cc:4985] TS e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): tablet ef445dad9acd4c69a9fb5d03dfe1df5f (table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]) successfully deleted
I20260204 02:53:34.228425 1413 ts_tablet_manager.cc:1434] T b8094eec37cd40b9b1f4931f24d824a8 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260204 02:53:34.229126 1435 ts_tablet_manager.cc:1916] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:34.231941 752 catalog_manager.cc:5654] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129). New cstate: current_term: 1 leader_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } health_report { overall_health: UNKNOWN } } }
I20260204 02:53:34.233917 1434 ts_tablet_manager.cc:1929] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260204 02:53:34.234061 1434 log.cc:1199] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/wal/wals/ef445dad9acd4c69a9fb5d03dfe1df5f
I20260204 02:53:34.234509 1434 ts_tablet_manager.cc:1950] T ef445dad9acd4c69a9fb5d03dfe1df5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting consensus metadata
I20260204 02:53:34.236028 1408 tablet_bootstrap.cc:492] T b8094eec37cd40b9b1f4931f24d824a8 P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:34.236203 1408 ts_tablet_manager.cc:1403] T b8094eec37cd40b9b1f4931f24d824a8 P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20260204 02:53:34.237030 1408 raft_consensus.cc:359] T b8094eec37cd40b9b1f4931f24d824a8 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.237222 1408 raft_consensus.cc:385] T b8094eec37cd40b9b1f4931f24d824a8 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.237299 1408 raft_consensus.cc:740] T b8094eec37cd40b9b1f4931f24d824a8 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.237555 1408 consensus_queue.cc:260] T b8094eec37cd40b9b1f4931f24d824a8 P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.238525 755 master_service.cc:954] ReplaceTablet: received request to replace tablet 64e25eca7e004205b329cb37a6e9471f from {username='slave'} at 127.0.0.1:38314
I20260204 02:53:34.240047 1408 ts_tablet_manager.cc:1434] T b8094eec37cd40b9b1f4931f24d824a8 P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260204 02:53:34.250242 744 catalog_manager.cc:4985] TS 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): tablet ef445dad9acd4c69a9fb5d03dfe1df5f (table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]) successfully deleted
I20260204 02:53:34.254205 1274 tablet_service.cc:1505] Processing CreateTablet for tablet b321632c62d545d9b3ce83683af6de42 (DEFAULT_TABLE table=default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.254622 1274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b321632c62d545d9b3ce83683af6de42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.257355 998 tablet_service.cc:1505] Processing CreateTablet for tablet b321632c62d545d9b3ce83683af6de42 (DEFAULT_TABLE table=default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.257802 998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b321632c62d545d9b3ce83683af6de42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.263360 755 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Sending DeleteTablet for 3 replicas of tablet 64e25eca7e004205b329cb37a6e9471f
I20260204 02:53:34.263549 1138 tablet_service.cc:1505] Processing CreateTablet for tablet b321632c62d545d9b3ce83683af6de42 (DEFAULT_TABLE table=default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.263958 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b321632c62d545d9b3ce83683af6de42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.263995 755 catalog_manager.cc:6724] ReplaceTablet: tablet 64e25eca7e004205b329cb37a6e9471f deleted and replaced by tablet 9a80cbfc27b54a5d945683c39020a027
I20260204 02:53:34.266392 999 tablet_service.cc:1552] Processing DeleteTablet for tablet 64e25eca7e004205b329cb37a6e9471f with delete_type TABLET_DATA_DELETED (replaced by tablet 9a80cbfc27b54a5d945683c39020a027) from {username='slave'} at 127.0.0.1:45802
I20260204 02:53:34.267298 1273 tablet_service.cc:1552] Processing DeleteTablet for tablet 64e25eca7e004205b329cb37a6e9471f with delete_type TABLET_DATA_DELETED (replaced by tablet 9a80cbfc27b54a5d945683c39020a027) from {username='slave'} at 127.0.0.1:42962
I20260204 02:53:34.267406 1434 tablet_replica.cc:333] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: stopping tablet replica
I20260204 02:53:34.267620 1434 raft_consensus.cc:2243] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260204 02:53:34.267797 1434 raft_consensus.cc:2272] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:34.268494 1434 ts_tablet_manager.cc:1916] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:34.271215 1408 tablet_bootstrap.cc:492] T b321632c62d545d9b3ce83683af6de42 P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:34.272281 755 master_service.cc:954] ReplaceTablet: received request to replace tablet 81d3882b863f4080a184bb084fcbff94 from {username='slave'} at 127.0.0.1:38314
I20260204 02:53:34.272939 1408 tablet_bootstrap.cc:654] T b321632c62d545d9b3ce83683af6de42 P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.273721 1409 tablet_bootstrap.cc:492] T b321632c62d545d9b3ce83683af6de42 P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:34.275141 1409 tablet_bootstrap.cc:654] T b321632c62d545d9b3ce83683af6de42 P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.277078 1413 tablet_bootstrap.cc:492] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:34.277896 1434 ts_tablet_manager.cc:1929] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260204 02:53:34.278041 1434 log.cc:1199] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/wal/wals/64e25eca7e004205b329cb37a6e9471f
I20260204 02:53:34.278354 1434 ts_tablet_manager.cc:1950] T 64e25eca7e004205b329cb37a6e9471f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting consensus metadata
I20260204 02:53:34.278584 1413 tablet_bootstrap.cc:654] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.279167 752 master_service.cc:954] ReplaceTablet: received request to replace tablet 81d3882b863f4080a184bb084fcbff94 from {username='slave'} at 127.0.0.1:38314
I20260204 02:53:34.279567 744 catalog_manager.cc:4985] TS 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): tablet 64e25eca7e004205b329cb37a6e9471f (table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]) successfully deleted
I20260204 02:53:34.286051 755 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Sending DeleteTablet for 3 replicas of tablet 81d3882b863f4080a184bb084fcbff94
I20260204 02:53:34.286152 1273 tablet_service.cc:1505] Processing CreateTablet for tablet 9a80cbfc27b54a5d945683c39020a027 (DEFAULT_TABLE table=default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.286665 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a80cbfc27b54a5d945683c39020a027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.286986 755 catalog_manager.cc:6724] ReplaceTablet: tablet 81d3882b863f4080a184bb084fcbff94 deleted and replaced by tablet 6f0852d9159a45a69935c76182584be3
I20260204 02:53:34.287961 1272 tablet_service.cc:1552] Processing DeleteTablet for tablet 81d3882b863f4080a184bb084fcbff94 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f0852d9159a45a69935c76182584be3) from {username='slave'} at 127.0.0.1:42962
I20260204 02:53:34.289036 1137 tablet_service.cc:1505] Processing CreateTablet for tablet 9a80cbfc27b54a5d945683c39020a027 (DEFAULT_TABLE table=default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.289501 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a80cbfc27b54a5d945683c39020a027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.290668 1436 tablet_replica.cc:333] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5: stopping tablet replica
I20260204 02:53:34.290859 1436 raft_consensus.cc:2243] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260204 02:53:34.291028 1436 raft_consensus.cc:2272] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:34.291972 1136 tablet_service.cc:1552] Processing DeleteTablet for tablet 81d3882b863f4080a184bb084fcbff94 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f0852d9159a45a69935c76182584be3) from {username='slave'} at 127.0.0.1:56478
I20260204 02:53:34.292606 999 tablet_service.cc:1552] Processing DeleteTablet for tablet 81d3882b863f4080a184bb084fcbff94 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f0852d9159a45a69935c76182584be3) from {username='slave'} at 127.0.0.1:45802
I20260204 02:53:34.292940 1436 ts_tablet_manager.cc:1916] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:34.293007 998 tablet_service.cc:1505] Processing CreateTablet for tablet 9a80cbfc27b54a5d945683c39020a027 (DEFAULT_TABLE table=default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.293329 998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a80cbfc27b54a5d945683c39020a027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.298007 1434 tablet_replica.cc:333] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: stopping tablet replica
I20260204 02:53:34.298192 1434 raft_consensus.cc:2243] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260204 02:53:34.298390 1434 raft_consensus.cc:2272] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:34.288110 752 master_service.cc:954] ReplaceTablet: received request to replace tablet 81d3882b863f4080a184bb084fcbff94 from {username='slave'} at 127.0.0.1:38314
I20260204 02:53:34.301594 1138 tablet_service.cc:1552] Processing DeleteTablet for tablet 64e25eca7e004205b329cb37a6e9471f with delete_type TABLET_DATA_DELETED (replaced by tablet 9a80cbfc27b54a5d945683c39020a027) from {username='slave'} at 127.0.0.1:56478
I20260204 02:53:34.302733 1434 ts_tablet_manager.cc:1916] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:34.308913 1413 tablet_bootstrap.cc:492] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:34.309072 1413 ts_tablet_manager.cc:1403] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.005s
I20260204 02:53:34.309171 1434 ts_tablet_manager.cc:1929] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260204 02:53:34.309275 1434 log.cc:1199] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/wal/wals/81d3882b863f4080a184bb084fcbff94
I20260204 02:53:34.309587 1434 ts_tablet_manager.cc:1950] T 81d3882b863f4080a184bb084fcbff94 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting consensus metadata
I20260204 02:53:34.309914 1413 raft_consensus.cc:359] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.310089 1413 raft_consensus.cc:385] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.310173 1413 raft_consensus.cc:740] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.310546 744 catalog_manager.cc:4985] TS 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): tablet 81d3882b863f4080a184bb084fcbff94 (table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]) successfully deleted
I20260204 02:53:34.310964 1413 consensus_queue.cc:260] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.311414 1413 ts_tablet_manager.cc:1434] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260204 02:53:34.311709 1413 tablet_bootstrap.cc:492] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:34.313309 1413 tablet_bootstrap.cc:654] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.314877 1409 tablet_bootstrap.cc:492] T b321632c62d545d9b3ce83683af6de42 P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:34.315012 1409 ts_tablet_manager.cc:1403] T b321632c62d545d9b3ce83683af6de42 P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.041s user 0.011s sys 0.001s
I20260204 02:53:34.315493 755 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_b896155e926a4679a475f25f74e33530"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:34.315742 1409 raft_consensus.cc:359] T b321632c62d545d9b3ce83683af6de42 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.315934 1409 raft_consensus.cc:385] T b321632c62d545d9b3ce83683af6de42 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.316016 1409 raft_consensus.cc:740] T b321632c62d545d9b3ce83683af6de42 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.316238 1409 consensus_queue.cc:260] T b321632c62d545d9b3ce83683af6de42 P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.317410 1436 ts_tablet_manager.cc:1929] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260204 02:53:34.317580 1436 log.cc:1199] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal/wals/64e25eca7e004205b329cb37a6e9471f
I20260204 02:53:34.317953 1436 ts_tablet_manager.cc:1950] T 64e25eca7e004205b329cb37a6e9471f P e2efdafb1246484a98dc5408be6746b5: Deleting consensus metadata
I20260204 02:53:34.318925 746 catalog_manager.cc:4985] TS e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): tablet 64e25eca7e004205b329cb37a6e9471f (table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]) successfully deleted
I20260204 02:53:34.319144 1435 ts_tablet_manager.cc:1929] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260204 02:53:34.319296 1435 log.cc:1199] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/wal/wals/ef445dad9acd4c69a9fb5d03dfe1df5f
I20260204 02:53:34.321213 1408 tablet_bootstrap.cc:492] T b321632c62d545d9b3ce83683af6de42 P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:34.323539 1273 tablet_service.cc:1505] Processing CreateTablet for tablet 6f0852d9159a45a69935c76182584be3 (DEFAULT_TABLE table=default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.323912 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f0852d9159a45a69935c76182584be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.324883 752 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_a2c8b05d2d724f50aaf92bddb8a266e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:34.326771 1137 tablet_service.cc:1505] Processing CreateTablet for tablet 6f0852d9159a45a69935c76182584be3 (DEFAULT_TABLE table=default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.327152 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f0852d9159a45a69935c76182584be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.329528 1408 ts_tablet_manager.cc:1403] T b321632c62d545d9b3ce83683af6de42 P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.010s
I20260204 02:53:34.329572 998 tablet_service.cc:1505] Processing CreateTablet for tablet 6f0852d9159a45a69935c76182584be3 (DEFAULT_TABLE table=default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.329936 998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f0852d9159a45a69935c76182584be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.330595 1408 raft_consensus.cc:359] T b321632c62d545d9b3ce83683af6de42 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.330778 1408 raft_consensus.cc:385] T b321632c62d545d9b3ce83683af6de42 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.330859 1408 raft_consensus.cc:740] T b321632c62d545d9b3ce83683af6de42 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.332051 1408 consensus_queue.cc:260] T b321632c62d545d9b3ce83683af6de42 P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.333082 1409 ts_tablet_manager.cc:1434] T b321632c62d545d9b3ce83683af6de42 P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20260204 02:53:34.335810 1409 tablet_bootstrap.cc:492] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:34.336061 1408 ts_tablet_manager.cc:1434] T b321632c62d545d9b3ce83683af6de42 P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260204 02:53:34.336694 1435 ts_tablet_manager.cc:1950] T ef445dad9acd4c69a9fb5d03dfe1df5f P 4dc2992e324049698330803c9ed2e3f8: Deleting consensus metadata
I20260204 02:53:34.337510 1409 tablet_bootstrap.cc:654] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.338440 1408 tablet_bootstrap.cc:492] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:34.338469 744 catalog_manager.cc:4985] TS 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): tablet ef445dad9acd4c69a9fb5d03dfe1df5f (table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]) successfully deleted
I20260204 02:53:34.338594 1436 tablet_replica.cc:333] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5: stopping tablet replica
I20260204 02:53:34.338781 1435 tablet_replica.cc:333] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8: stopping tablet replica
I20260204 02:53:34.338806 1436 raft_consensus.cc:2243] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260204 02:53:34.338972 1436 raft_consensus.cc:2272] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:34.339002 1435 raft_consensus.cc:2243] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [term 1 LEADER]: Raft consensus shutting down.
I20260204 02:53:34.339226 1435 pending_rounds.cc:70] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8: Trying to abort 1 pending ops.
I20260204 02:53:34.339282 1435 pending_rounds.cc:77] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7250631123387375616 op_type: NO_OP noop_request { }
I20260204 02:53:34.339398 1435 raft_consensus.cc:2889] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260204 02:53:34.339510 1435 raft_consensus.cc:2272] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:34.340070 1436 ts_tablet_manager.cc:1916] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:34.340627 1435 ts_tablet_manager.cc:1916] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:34.341717 1408 tablet_bootstrap.cc:654] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.344064 1413 tablet_bootstrap.cc:492] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:34.344220 1413 ts_tablet_manager.cc:1403] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.003s
I20260204 02:53:34.344957 1413 raft_consensus.cc:359] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.345153 1413 raft_consensus.cc:385] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.345228 1413 raft_consensus.cc:740] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.345479 1413 consensus_queue.cc:260] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.346606 1436 ts_tablet_manager.cc:1929] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260204 02:53:34.346751 1435 ts_tablet_manager.cc:1929] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260204 02:53:34.346789 1436 log.cc:1199] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal/wals/81d3882b863f4080a184bb084fcbff94
I20260204 02:53:34.346925 1435 log.cc:1199] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/wal/wals/81d3882b863f4080a184bb084fcbff94
I20260204 02:53:34.347268 1436 ts_tablet_manager.cc:1950] T 81d3882b863f4080a184bb084fcbff94 P e2efdafb1246484a98dc5408be6746b5: Deleting consensus metadata
I20260204 02:53:34.347385 1435 ts_tablet_manager.cc:1950] T 81d3882b863f4080a184bb084fcbff94 P 4dc2992e324049698330803c9ed2e3f8: Deleting consensus metadata
I20260204 02:53:34.348245 746 catalog_manager.cc:4985] TS e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): tablet 81d3882b863f4080a184bb084fcbff94 (table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]) successfully deleted
I20260204 02:53:34.350521 753 catalog_manager.cc:2174] created HMS catalog entry for table [id=b27e50e8b8b84fd2972f1b7c8768dd29]
I20260204 02:53:34.351426 1413 ts_tablet_manager.cc:1434] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260204 02:53:34.351577 744 catalog_manager.cc:4985] TS 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): tablet 81d3882b863f4080a184bb084fcbff94 (table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]) successfully deleted
I20260204 02:53:34.352869 1408 tablet_bootstrap.cc:492] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:34.353523 1413 tablet_bootstrap.cc:492] T 6f0852d9159a45a69935c76182584be3 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:34.355103 1413 tablet_bootstrap.cc:654] T 6f0852d9159a45a69935c76182584be3 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.355930 1409 tablet_bootstrap.cc:492] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:34.356103 1409 ts_tablet_manager.cc:1403] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20260204 02:53:34.356998 1409 raft_consensus.cc:359] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.357203 1409 raft_consensus.cc:385] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.357851 1409 raft_consensus.cc:740] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.358105 1409 consensus_queue.cc:260] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.359949 1409 ts_tablet_manager.cc:1434] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260204 02:53:34.361214 1409 tablet_bootstrap.cc:492] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:34.362800 1409 tablet_bootstrap.cc:654] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.365885 998 tablet_service.cc:1505] Processing CreateTablet for tablet 8c77d21f66a640d88bf1865d6176c969 (DEFAULT_TABLE table=default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.366281 998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c77d21f66a640d88bf1865d6176c969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.366470 1273 tablet_service.cc:1505] Processing CreateTablet for tablet 8c77d21f66a640d88bf1865d6176c969 (DEFAULT_TABLE table=default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.367002 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c77d21f66a640d88bf1865d6176c969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.372695 1137 tablet_service.cc:1505] Processing CreateTablet for tablet 8c77d21f66a640d88bf1865d6176c969 (DEFAULT_TABLE table=default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.373052 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c77d21f66a640d88bf1865d6176c969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.380306 1413 tablet_bootstrap.cc:492] T 6f0852d9159a45a69935c76182584be3 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:34.380460 1413 ts_tablet_manager.cc:1403] T 6f0852d9159a45a69935c76182584be3 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.008s
I20260204 02:53:34.381233 1413 raft_consensus.cc:359] T 6f0852d9159a45a69935c76182584be3 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.381448 1413 raft_consensus.cc:385] T 6f0852d9159a45a69935c76182584be3 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.381541 1413 raft_consensus.cc:740] T 6f0852d9159a45a69935c76182584be3 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.381778 1413 consensus_queue.cc:260] T 6f0852d9159a45a69935c76182584be3 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.383675 1413 ts_tablet_manager.cc:1434] T 6f0852d9159a45a69935c76182584be3 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260204 02:53:34.384893 1413 tablet_bootstrap.cc:492] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:34.386538 1413 tablet_bootstrap.cc:654] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.386915 753 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_bdc0d01a86b043beb3441b2078794538"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:34.387401 1435 tablet_replica.cc:333] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8: stopping tablet replica
I20260204 02:53:34.387646 1435 raft_consensus.cc:2243] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260204 02:53:34.387873 1435 raft_consensus.cc:2272] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:34.353003 1408 ts_tablet_manager.cc:1403] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20260204 02:53:34.388440 1413 tablet_bootstrap.cc:492] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:34.388590 1413 ts_tablet_manager.cc:1403] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260204 02:53:34.388989 1408 raft_consensus.cc:359] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.389138 1413 raft_consensus.cc:359] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:34.389258 1408 raft_consensus.cc:385] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.389320 1413 raft_consensus.cc:385] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.389346 1408 raft_consensus.cc:740] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.389400 1413 raft_consensus.cc:740] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.389617 1413 consensus_queue.cc:260] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:34.390025 1435 ts_tablet_manager.cc:1916] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:34.390779 1413 ts_tablet_manager.cc:1434] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260204 02:53:34.391639 1408 consensus_queue.cc:260] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.394565 1408 ts_tablet_manager.cc:1434] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260204 02:53:34.395687 1408 tablet_bootstrap.cc:492] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:34.397722 1408 tablet_bootstrap.cc:654] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.403445 1435 ts_tablet_manager.cc:1929] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260204 02:53:34.403620 1435 log.cc:1199] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/wal/wals/64e25eca7e004205b329cb37a6e9471f
I20260204 02:53:34.403983 1435 ts_tablet_manager.cc:1950] T 64e25eca7e004205b329cb37a6e9471f P 4dc2992e324049698330803c9ed2e3f8: Deleting consensus metadata
I20260204 02:53:34.405089 744 catalog_manager.cc:4985] TS 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): tablet 64e25eca7e004205b329cb37a6e9471f (table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]) successfully deleted
I20260204 02:53:34.406490 1408 tablet_bootstrap.cc:492] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:34.406649 1408 ts_tablet_manager.cc:1403] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20260204 02:53:34.407455 1408 raft_consensus.cc:359] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.407680 1408 raft_consensus.cc:385] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.407792 1408 raft_consensus.cc:740] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.408067 1408 consensus_queue.cc:260] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.411893 1409 tablet_bootstrap.cc:492] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:34.412048 1409 ts_tablet_manager.cc:1403] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.003s
I20260204 02:53:34.412779 1409 raft_consensus.cc:359] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.412959 1409 raft_consensus.cc:385] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.413049 1409 raft_consensus.cc:740] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.413285 1409 consensus_queue.cc:260] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.413771 1409 ts_tablet_manager.cc:1434] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260204 02:53:34.414175 1408 ts_tablet_manager.cc:1434] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260204 02:53:34.416945 1408 tablet_bootstrap.cc:492] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:34.418673 1408 tablet_bootstrap.cc:654] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.419466 1409 tablet_bootstrap.cc:492] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:34.421538 1409 tablet_bootstrap.cc:654] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.434398 1409 tablet_bootstrap.cc:492] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:34.434590 1409 ts_tablet_manager.cc:1403] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.001s
I20260204 02:53:34.435387 1409 raft_consensus.cc:359] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:34.435634 1409 raft_consensus.cc:385] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.435731 1409 raft_consensus.cc:740] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.436067 1409 consensus_queue.cc:260] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:34.440714 1409 ts_tablet_manager.cc:1434] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260204 02:53:34.440857 758 catalog_manager.cc:2174] created HMS catalog entry for table [id=4b389805ed274ba89b2f4fa10039d793]
I20260204 02:53:34.445966 758 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:38314 (request call id 4) took 1020 ms. Trace:
I20260204 02:53:34.446139 758 rpcz_store.cc:276] 0204 02:53:33.425280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0204 02:53:33.425327 (+ 47us) service_pool.cc:225] Handling call
0204 02:53:33.744853 (+319526us) catalog_manager.cc:2101] Acquired catalog manager lock
0204 02:53:33.745335 (+ 482us) catalog_manager.cc:2147] Created new table and tablet info
0204 02:53:34.440801 (+695466us) catalog_manager.cc:2173] Created new table in HMS catalog
0204 02:53:34.445634 (+ 4833us) catalog_manager.cc:2207] Wrote table and tablets to system table
0204 02:53:34.445791 (+ 157us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0204 02:53:34.445912 (+ 121us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0204 02:53:34.441571 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0204 02:53:34.441712 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
0204 02:53:34.441720 (+ 8us) write_op.cc:435] Acquired schema lock
0204 02:53:34.441723 (+ 3us) tablet.cc:660] Decoding operations
0204 02:53:34.441827 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0204 02:53:34.441842 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0204 02:53:34.441849 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0204 02:53:34.441940 (+ 91us) tablet.cc:699] Row locks acquired
0204 02:53:34.441943 (+ 3us) write_op.cc:260] PREPARE: finished
0204 02:53:34.441988 (+ 45us) write_op.cc:270] Start()
0204 02:53:34.442028 (+ 40us) write_op.cc:276] Timestamp: P: 1770173614441969 usec, L: 0
0204 02:53:34.442032 (+ 4us) op_driver.cc:348] REPLICATION: starting
0204 02:53:34.442242 (+ 210us) log.cc:844] Serialized 432 byte log entry
0204 02:53:34.445088 (+ 2846us) op_driver.cc:464] REPLICATION: finished
0204 02:53:34.445202 (+ 114us) write_op.cc:301] APPLY: starting
0204 02:53:34.445215 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0204 02:53:34.445259 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0204 02:53:34.445281 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0204 02:53:34.445370 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
0204 02:53:34.445400 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0204 02:53:34.445406 (+ 6us) write_op.cc:312] APPLY: finished
0204 02:53:34.445475 (+ 69us) log.cc:844] Serialized 28 byte log entry
0204 02:53:34.445527 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0204 02:53:34.445546 (+ 19us) write_op.cc:454] Released schema lock
0204 02:53:34.445552 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0204 02:53:34.445620 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0204 02:53:34.445624 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":604992,"Hive Metastore.run_cpu_time_us":293,"Hive Metastore.run_wall_time_us":90298,"child_traces":[["op",{"apply.queue_time_us":73,"apply.run_cpu_time_us":411,"apply.run_wall_time_us":410,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":795,"prepare.run_wall_time_us":794,"raft.queue_time_us":445,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":266,"replication_time_us":3029}]]}
I20260204 02:53:34.448933 1416 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:34.449112 1416 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.449747 1416 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:34.450284 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a80cbfc27b54a5d945683c39020a027" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5" is_pre_election: true
I20260204 02:53:34.450296 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a80cbfc27b54a5d945683c39020a027" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8" is_pre_election: true
I20260204 02:53:34.450493 1294 raft_consensus.cc:2468] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 0.
I20260204 02:53:34.450493 1158 raft_consensus.cc:2468] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 0.
I20260204 02:53:34.450896 952 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:34.451251 1416 raft_consensus.cc:2804] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:34.451385 1416 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:34.451508 1416 raft_consensus.cc:3060] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.455260 1137 tablet_service.cc:1505] Processing CreateTablet for tablet 8bbcb21479124152831cf7d022fa4e5f (DEFAULT_TABLE table=default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.455513 1408 tablet_bootstrap.cc:492] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:34.455681 1408 ts_tablet_manager.cc:1403] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.019s
I20260204 02:53:34.455744 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bbcb21479124152831cf7d022fa4e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.455965 758 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_e971e55de1b44441b34f0913c0d665a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:34.456525 1408 raft_consensus.cc:359] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:34.456734 1408 raft_consensus.cc:385] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.456816 1408 raft_consensus.cc:740] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.459887 1273 tablet_service.cc:1505] Processing CreateTablet for tablet 8bbcb21479124152831cf7d022fa4e5f (DEFAULT_TABLE table=default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.460254 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bbcb21479124152831cf7d022fa4e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.460693 1408 consensus_queue.cc:260] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:34.462859 1408 ts_tablet_manager.cc:1434] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260204 02:53:34.464550 998 tablet_service.cc:1505] Processing CreateTablet for tablet 8bbcb21479124152831cf7d022fa4e5f (DEFAULT_TABLE table=default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.464946 998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bbcb21479124152831cf7d022fa4e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.469967 1416 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.470827 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a80cbfc27b54a5d945683c39020a027" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8"
I20260204 02:53:34.471021 1158 raft_consensus.cc:3060] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.473450 1158 raft_consensus.cc:2468] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 1.
I20260204 02:53:34.474076 952 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:34.474419 1433 raft_consensus.cc:2804] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:34.483263 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a80cbfc27b54a5d945683c39020a027" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5"
I20260204 02:53:34.483516 1294 raft_consensus.cc:3060] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.485987 1294 raft_consensus.cc:2468] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 1.
I20260204 02:53:34.486722 1416 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 election: Requested vote from peers 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:34.488194 1409 tablet_bootstrap.cc:492] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:34.488692 1408 tablet_bootstrap.cc:492] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:34.489730 1409 tablet_bootstrap.cc:654] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.490162 1408 tablet_bootstrap.cc:654] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.496534 1433 raft_consensus.cc:697] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Running, Role: LEADER
I20260204 02:53:34.496995 1433 consensus_queue.cc:237] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.502873 756 catalog_manager.cc:2174] created HMS catalog entry for table [id=5705cdf41ef94c8d94f3b3db54323796]
I20260204 02:53:34.508664 756 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:38314 (request call id 6) took 1082 ms. Trace:
I20260204 02:53:34.508781 756 rpcz_store.cc:276] 0204 02:53:33.425928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0204 02:53:33.425979 (+ 51us) service_pool.cc:225] Handling call
0204 02:53:33.745889 (+319910us) catalog_manager.cc:2101] Acquired catalog manager lock
0204 02:53:33.746439 (+ 550us) catalog_manager.cc:2147] Created new table and tablet info
0204 02:53:34.502814 (+756375us) catalog_manager.cc:2173] Created new table in HMS catalog
0204 02:53:34.508419 (+ 5605us) catalog_manager.cc:2207] Wrote table and tablets to system table
0204 02:53:34.508524 (+ 105us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0204 02:53:34.508625 (+ 101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0204 02:53:34.503788 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0204 02:53:34.503927 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
0204 02:53:34.503933 (+ 6us) write_op.cc:435] Acquired schema lock
0204 02:53:34.503935 (+ 2us) tablet.cc:660] Decoding operations
0204 02:53:34.504023 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0204 02:53:34.504039 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0204 02:53:34.504044 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0204 02:53:34.504138 (+ 94us) tablet.cc:699] Row locks acquired
0204 02:53:34.504142 (+ 4us) write_op.cc:260] PREPARE: finished
0204 02:53:34.504176 (+ 34us) write_op.cc:270] Start()
0204 02:53:34.504210 (+ 34us) write_op.cc:276] Timestamp: P: 1770173614504160 usec, L: 0
0204 02:53:34.504213 (+ 3us) op_driver.cc:348] REPLICATION: starting
0204 02:53:34.504393 (+ 180us) log.cc:844] Serialized 432 byte log entry
0204 02:53:34.507124 (+ 2731us) op_driver.cc:464] REPLICATION: finished
0204 02:53:34.507254 (+ 130us) write_op.cc:301] APPLY: starting
0204 02:53:34.507269 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0204 02:53:34.507462 (+ 193us) tablet.cc:1368] finished BulkCheckPresence
0204 02:53:34.507528 (+ 66us) tablet.cc:1370] starting ApplyRowOperation cycle
0204 02:53:34.507658 (+ 130us) tablet.cc:1381] finished ApplyRowOperation cycle
0204 02:53:34.507690 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0204 02:53:34.507697 (+ 7us) write_op.cc:312] APPLY: finished
0204 02:53:34.507765 (+ 68us) log.cc:844] Serialized 28 byte log entry
0204 02:53:34.507819 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0204 02:53:34.507847 (+ 28us) write_op.cc:454] Released schema lock
0204 02:53:34.507853 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0204 02:53:34.507913 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0204 02:53:34.507916 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":694186,"Hive Metastore.run_cpu_time_us":232,"Hive Metastore.run_wall_time_us":61725,"child_traces":[["op",{"apply.queue_time_us":86,"apply.run_cpu_time_us":496,"apply.run_wall_time_us":652,"num_ops":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":713,"prepare.run_wall_time_us":713,"raft.queue_time_us":393,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":240,"replication_time_us":2884}]]}
I20260204 02:53:34.513041 1413 tablet_bootstrap.cc:492] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:34.514561 1413 tablet_bootstrap.cc:654] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.518900 1137 tablet_service.cc:1505] Processing CreateTablet for tablet 8f1bc638fe8e48189c15d18209efdd1d (DEFAULT_TABLE table=default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.519229 999 tablet_service.cc:1505] Processing CreateTablet for tablet 8f1bc638fe8e48189c15d18209efdd1d (DEFAULT_TABLE table=default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.519302 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1bc638fe8e48189c15d18209efdd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.519618 999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1bc638fe8e48189c15d18209efdd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.522500 1273 tablet_service.cc:1505] Processing CreateTablet for tablet 8f1bc638fe8e48189c15d18209efdd1d (DEFAULT_TABLE table=default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.522910 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1bc638fe8e48189c15d18209efdd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.532407 756 catalog_manager.cc:5654] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129). New cstate: current_term: 1 leader_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } health_report { overall_health: UNKNOWN } } }
I20260204 02:53:34.542269 756 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38314:
name: "default.Table_237c40ddb38c410c9ce533b72060712f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:34.545190 1413 tablet_bootstrap.cc:492] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:34.545249 1408 tablet_bootstrap.cc:492] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:34.545377 1413 ts_tablet_manager.cc:1403] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.003s
I20260204 02:53:34.545569 1408 ts_tablet_manager.cc:1403] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.057s user 0.030s sys 0.011s
I20260204 02:53:34.546366 1413 raft_consensus.cc:359] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:34.546569 1413 raft_consensus.cc:385] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.546664 1413 raft_consensus.cc:740] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.546473 1408 raft_consensus.cc:359] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:34.546895 1408 raft_consensus.cc:385] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.546939 1413 consensus_queue.cc:260] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:34.547030 1408 raft_consensus.cc:740] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.547349 1408 consensus_queue.cc:260] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:34.547952 1408 ts_tablet_manager.cc:1434] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260204 02:53:34.548377 1408 tablet_bootstrap.cc:492] T 8f1bc638fe8e48189c15d18209efdd1d P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:34.550117 1408 tablet_bootstrap.cc:654] T 8f1bc638fe8e48189c15d18209efdd1d P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.558004 1413 ts_tablet_manager.cc:1434] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20260204 02:53:34.558387 1409 tablet_bootstrap.cc:492] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:34.558555 1409 ts_tablet_manager.cc:1403] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.009s
I20260204 02:53:34.562103 1413 tablet_bootstrap.cc:492] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:34.563807 1413 tablet_bootstrap.cc:654] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.559271 1409 raft_consensus.cc:359] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:34.565009 1409 raft_consensus.cc:385] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.565130 1409 raft_consensus.cc:740] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.565383 1409 consensus_queue.cc:260] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:34.569535 1409 ts_tablet_manager.cc:1434] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20260204 02:53:34.572834 1409 tablet_bootstrap.cc:492] T 8f1bc638fe8e48189c15d18209efdd1d P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:34.574805 1409 tablet_bootstrap.cc:654] T 8f1bc638fe8e48189c15d18209efdd1d P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.577178 1413 tablet_bootstrap.cc:492] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:34.577350 1413 ts_tablet_manager.cc:1403] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20260204 02:53:34.578097 1413 raft_consensus.cc:359] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:34.578289 1413 raft_consensus.cc:385] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.578365 1413 raft_consensus.cc:740] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.578629 1413 consensus_queue.cc:260] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:34.578905 1433 raft_consensus.cc:493] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:34.579041 1433 raft_consensus.cc:515] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.579969 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b321632c62d545d9b3ce83683af6de42" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8" is_pre_election: true
I20260204 02:53:34.580149 1158 raft_consensus.cc:2468] T b321632c62d545d9b3ce83683af6de42 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 0.
I20260204 02:53:34.580615 952 leader_election.cc:304] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:34.580868 1416 raft_consensus.cc:2804] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:34.580973 1416 raft_consensus.cc:493] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:34.581053 1416 raft_consensus.cc:3060] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.583457 1416 raft_consensus.cc:515] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.584607 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b321632c62d545d9b3ce83683af6de42" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5"
I20260204 02:53:34.584820 1294 raft_consensus.cc:3060] T b321632c62d545d9b3ce83683af6de42 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.587953 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b321632c62d545d9b3ce83683af6de42" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8"
I20260204 02:53:34.588122 1158 raft_consensus.cc:3060] T b321632c62d545d9b3ce83683af6de42 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.590893 1433 leader_election.cc:290] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:34.591686 1293 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b321632c62d545d9b3ce83683af6de42" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5" is_pre_election: true
I20260204 02:53:34.592312 1158 raft_consensus.cc:2468] T b321632c62d545d9b3ce83683af6de42 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 1.
I20260204 02:53:34.593282 952 leader_election.cc:304] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:34.593583 1433 raft_consensus.cc:2804] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:34.593722 1433 raft_consensus.cc:697] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Running, Role: LEADER
I20260204 02:53:34.594007 1433 consensus_queue.cc:237] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.595151 1294 raft_consensus.cc:2468] T b321632c62d545d9b3ce83683af6de42 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 1.
I20260204 02:53:34.595638 1416 leader_election.cc:290] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 election: Requested vote from peers 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:34.597013 1409 tablet_bootstrap.cc:492] T 8f1bc638fe8e48189c15d18209efdd1d P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:34.597185 1409 ts_tablet_manager.cc:1403] T 8f1bc638fe8e48189c15d18209efdd1d P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20260204 02:53:34.598126 1409 raft_consensus.cc:359] T 8f1bc638fe8e48189c15d18209efdd1d P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:34.598574 1413 ts_tablet_manager.cc:1434] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20260204 02:53:34.598814 1409 raft_consensus.cc:385] T 8f1bc638fe8e48189c15d18209efdd1d P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.598912 1409 raft_consensus.cc:740] T 8f1bc638fe8e48189c15d18209efdd1d P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.599186 1409 consensus_queue.cc:260] T 8f1bc638fe8e48189c15d18209efdd1d P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:34.602326 1409 ts_tablet_manager.cc:1434] T 8f1bc638fe8e48189c15d18209efdd1d P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260204 02:53:34.602711 1408 tablet_bootstrap.cc:492] T 8f1bc638fe8e48189c15d18209efdd1d P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:34.602938 1408 ts_tablet_manager.cc:1403] T 8f1bc638fe8e48189c15d18209efdd1d P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.055s user 0.015s sys 0.035s
I20260204 02:53:34.603714 1408 raft_consensus.cc:359] T 8f1bc638fe8e48189c15d18209efdd1d P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:34.603964 1408 raft_consensus.cc:385] T 8f1bc638fe8e48189c15d18209efdd1d P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.604068 1408 raft_consensus.cc:740] T 8f1bc638fe8e48189c15d18209efdd1d P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.604347 1408 consensus_queue.cc:260] T 8f1bc638fe8e48189c15d18209efdd1d P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:34.605823 1408 ts_tablet_manager.cc:1434] T 8f1bc638fe8e48189c15d18209efdd1d P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260204 02:53:34.609930 1412 raft_consensus.cc:493] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:34.610085 751 catalog_manager.cc:2174] created HMS catalog entry for table [id=ceb020292ca6471fb47601b65b5ef0d5]
I20260204 02:53:34.610064 1412 raft_consensus.cc:515] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.610610 1412 leader_election.cc:290] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:34.611737 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8094eec37cd40b9b1f4931f24d824a8" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5" is_pre_election: true
I20260204 02:53:34.611963 1294 raft_consensus.cc:2468] T b8094eec37cd40b9b1f4931f24d824a8 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 0.
I20260204 02:53:34.612361 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8094eec37cd40b9b1f4931f24d824a8" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:34.612565 1019 raft_consensus.cc:2468] T b8094eec37cd40b9b1f4931f24d824a8 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 0.
I20260204 02:53:34.612972 1093 leader_election.cc:304] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:34.613209 1412 raft_consensus.cc:2804] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:34.613299 1412 raft_consensus.cc:493] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:34.613364 1412 raft_consensus.cc:3060] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.615898 1412 raft_consensus.cc:515] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.617213 751 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:38314 (request call id 11) took 1188 ms. Trace:
I20260204 02:53:34.617311 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8094eec37cd40b9b1f4931f24d824a8" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
I20260204 02:53:34.617497 1019 raft_consensus.cc:3060] T b8094eec37cd40b9b1f4931f24d824a8 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.617363 751 rpcz_store.cc:276] 0204 02:53:33.428872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0204 02:53:33.428919 (+ 47us) service_pool.cc:225] Handling call
0204 02:53:33.747905 (+318986us) catalog_manager.cc:2101] Acquired catalog manager lock
0204 02:53:33.748310 (+ 405us) catalog_manager.cc:2147] Created new table and tablet info
0204 02:53:34.610036 (+861726us) catalog_manager.cc:2173] Created new table in HMS catalog
0204 02:53:34.616902 (+ 6866us) catalog_manager.cc:2207] Wrote table and tablets to system table
0204 02:53:34.617043 (+ 141us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0204 02:53:34.617168 (+ 125us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0204 02:53:34.610782 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0204 02:53:34.610911 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0204 02:53:34.610917 (+ 6us) write_op.cc:435] Acquired schema lock
0204 02:53:34.610919 (+ 2us) tablet.cc:660] Decoding operations
0204 02:53:34.611018 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0204 02:53:34.611031 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0204 02:53:34.611036 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0204 02:53:34.611110 (+ 74us) tablet.cc:699] Row locks acquired
0204 02:53:34.611112 (+ 2us) write_op.cc:260] PREPARE: finished
0204 02:53:34.611151 (+ 39us) write_op.cc:270] Start()
0204 02:53:34.611185 (+ 34us) write_op.cc:276] Timestamp: P: 1770173614611135 usec, L: 0
0204 02:53:34.611188 (+ 3us) op_driver.cc:348] REPLICATION: starting
0204 02:53:34.611364 (+ 176us) log.cc:844] Serialized 432 byte log entry
0204 02:53:34.615399 (+ 4035us) op_driver.cc:464] REPLICATION: finished
0204 02:53:34.615546 (+ 147us) write_op.cc:301] APPLY: starting
0204 02:53:34.615579 (+ 33us) tablet.cc:1365] starting BulkCheckPresence
0204 02:53:34.615634 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0204 02:53:34.615658 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0204 02:53:34.615757 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
0204 02:53:34.615791 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0204 02:53:34.615798 (+ 7us) write_op.cc:312] APPLY: finished
0204 02:53:34.615885 (+ 87us) log.cc:844] Serialized 28 byte log entry
0204 02:53:34.615963 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0204 02:53:34.615985 (+ 22us) write_op.cc:454] Released schema lock
0204 02:53:34.615993 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0204 02:53:34.616091 (+ 98us) write_op.cc:489] Releasing partition, row and schema locks
0204 02:53:34.616097 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":754189,"Hive Metastore.run_cpu_time_us":322,"Hive Metastore.run_wall_time_us":107264,"child_traces":[["op",{"apply.queue_time_us":98,"apply.run_cpu_time_us":531,"apply.run_wall_time_us":530,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":685,"prepare.run_wall_time_us":686,"raft.queue_time_us":1211,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":290,"replication_time_us":4178}]]}
I20260204 02:53:34.623092 1019 raft_consensus.cc:2468] T b8094eec37cd40b9b1f4931f24d824a8 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
I20260204 02:53:34.623883 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8094eec37cd40b9b1f4931f24d824a8" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5"
I20260204 02:53:34.624073 1294 raft_consensus.cc:3060] T b8094eec37cd40b9b1f4931f24d824a8 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.624763 1091 leader_election.cc:304] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:34.627704 1294 raft_consensus.cc:2468] T b8094eec37cd40b9b1f4931f24d824a8 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
I20260204 02:53:34.628119 1412 leader_election.cc:290] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 election: Requested vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:34.628293 1412 raft_consensus.cc:2804] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:34.628403 1412 raft_consensus.cc:697] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Running, Role: LEADER
I20260204 02:53:34.628628 1412 consensus_queue.cc:237] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.634613 1273 tablet_service.cc:1505] Processing CreateTablet for tablet 2b883ef926504d489f3a1c0aec6db438 (DEFAULT_TABLE table=default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.634760 998 tablet_service.cc:1505] Processing CreateTablet for tablet 2b883ef926504d489f3a1c0aec6db438 (DEFAULT_TABLE table=default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.635092 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b883ef926504d489f3a1c0aec6db438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.635181 998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b883ef926504d489f3a1c0aec6db438. 1 dirs total, 0 dirs full, 0 dirs failed
F20260204 02:53:34.640262 922 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_3a403dbfa3044e55abc223b3acd75b6b, message={"server":"","servicePrincipal":"","db":"default","table":"table_3a403dbfa3044e55abc223b3acd75b6b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_3a403dbfa3044e55abc223b3acd75b6b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_3a403dbfa3044e55abc223b3acd75b6b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_3a403dbfa3044e55abc223b3acd75b6b\",\"kudu.table_id\":\"333921075666492fb4e06fa35e04487b\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_0312705ce5e140d4bcf88a92cab2bb7b, message={"server":"","servicePrincipal":"","db":"default","table":"table_0312705ce5e140d4bcf88a92cab2bb7b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0312705ce5e140d4bcf88a92cab2bb7b\",\"kudu.table_id\":\"8df7ca3e2276436e8bdb4480fc3dd711\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20260204 02:53:34.641827 1413 tablet_bootstrap.cc:492] T 2b883ef926504d489f3a1c0aec6db438 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:34.642021 1408 tablet_bootstrap.cc:492] T 2b883ef926504d489f3a1c0aec6db438 P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
@ 0x7fac4b655dcd google::LogMessage::Fail() at ??:0
I20260204 02:53:34.643456 1408 tablet_bootstrap.cc:654] T 2b883ef926504d489f3a1c0aec6db438 P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:34.643745 1413 tablet_bootstrap.cc:654] T 2b883ef926504d489f3a1c0aec6db438 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
@ 0x7fac4b659b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fac4b6557cc google::LogMessage::Flush() at ??:0
I20260204 02:53:34.650975 751 catalog_manager.cc:5654] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129). New cstate: current_term: 1 leader_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } health_report { overall_health: UNKNOWN } } }
@ 0x7fac4b655a11 google::LogMessage::~LogMessage() at ??:0
I20260204 02:53:34.668371 1413 tablet_bootstrap.cc:492] T 2b883ef926504d489f3a1c0aec6db438 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:34.668630 1413 ts_tablet_manager.cc:1403] T 2b883ef926504d489f3a1c0aec6db438 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.005s
I20260204 02:53:34.669489 1413 raft_consensus.cc:359] T 2b883ef926504d489f3a1c0aec6db438 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.669714 1413 raft_consensus.cc:385] T 2b883ef926504d489f3a1c0aec6db438 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.669800 1413 raft_consensus.cc:740] T 2b883ef926504d489f3a1c0aec6db438 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.670079 1413 consensus_queue.cc:260] T 2b883ef926504d489f3a1c0aec6db438 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.672349 1413 ts_tablet_manager.cc:1434] T 2b883ef926504d489f3a1c0aec6db438 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260204 02:53:34.678871 751 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38314:
table { table_name: "default.Table_3d3a816d407e429ea458b787f879ede0" } new_table_name: "default.Table_541e82c4f45e4e74926e15734baef2cd" modify_external_catalogs: true
I20260204 02:53:34.682665 1408 tablet_bootstrap.cc:492] T 2b883ef926504d489f3a1c0aec6db438 P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:34.682840 1408 ts_tablet_manager.cc:1403] T 2b883ef926504d489f3a1c0aec6db438 P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.008s
@ 0x7fac620897f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
W20260204 02:53:34.683312 1091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.129:45767, user_credentials={real_user=slave}} blocked reactor thread for 58580us
I20260204 02:53:34.683797 1408 raft_consensus.cc:359] T 2b883ef926504d489f3a1c0aec6db438 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.684000 1408 raft_consensus.cc:385] T 2b883ef926504d489f3a1c0aec6db438 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.684084 1408 raft_consensus.cc:740] T 2b883ef926504d489f3a1c0aec6db438 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:34.684067 1137 tablet_service.cc:1505] Processing CreateTablet for tablet 2b883ef926504d489f3a1c0aec6db438 (DEFAULT_TABLE table=default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:34.684456 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b883ef926504d489f3a1c0aec6db438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:34.684458 1408 consensus_queue.cc:260] T 2b883ef926504d489f3a1c0aec6db438 P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.685071 1408 ts_tablet_manager.cc:1434] T 2b883ef926504d489f3a1c0aec6db438 P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
@ 0x7fac620880a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260204 02:53:34.692571 1409 tablet_bootstrap.cc:492] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:34.694165 1409 tablet_bootstrap.cc:654] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
@ 0x7fac6208fd1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260204 02:53:34.696316 1409 tablet_bootstrap.cc:492] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:34.696460 1409 ts_tablet_manager.cc:1403] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260204 02:53:34.697261 1409 raft_consensus.cc:359] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.697485 1409 raft_consensus.cc:385] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:34.697570 1409 raft_consensus.cc:740] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
@ 0x7fac6208fb99 std::_Function_handler<>::_M_invoke() at ??:0
I20260204 02:53:34.697809 1409 consensus_queue.cc:260] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.698367 1409 ts_tablet_manager.cc:1434] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7fac4e3caf99 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fac519d76db start_thread at ??:0
I20260204 02:53:34.706875 1417 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:34.707072 1417 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
@ 0x7fac4a50671f clone at ??:0
I20260204 02:53:34.709053 1417 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:34.730074 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c77d21f66a640d88bf1865d6176c969" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8" is_pre_election: true
I20260204 02:53:34.730325 1158 raft_consensus.cc:2468] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 0.
I20260204 02:53:34.730607 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c77d21f66a640d88bf1865d6176c969" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:34.730851 1019 raft_consensus.cc:2468] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 0.
I20260204 02:53:34.730873 1227 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:34.731261 1417 raft_consensus.cc:2804] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:34.731375 1417 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:34.731521 1417 raft_consensus.cc:3060] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.735002 1417 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:34.736050 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c77d21f66a640d88bf1865d6176c969" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
I20260204 02:53:34.736275 1019 raft_consensus.cc:3060] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.739167 1019 raft_consensus.cc:2468] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 1.
I20260204 02:53:34.739620 1227 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:34.739969 1417 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 election: Requested vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:34.740186 1417 raft_consensus.cc:2804] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:34.740315 1417 raft_consensus.cc:697] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [term 1 LEADER]: Becoming Leader. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Running, Role: LEADER
I20260204 02:53:34.740734 1417 consensus_queue.cc:237] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:34.758560 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c77d21f66a640d88bf1865d6176c969" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8"
I20260204 02:53:34.758844 1158 raft_consensus.cc:3060] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.761565 1158 raft_consensus.cc:2468] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 1.
W20260204 02:53:34.848510 1429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:53:34.849368 1429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:53:34.849484 1429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260204 02:53:34.861428 1429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260204 02:53:34.861572 1429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:53:34.861677 1429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260204 02:53:34.861749 1429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260204 02:53:34.877848 1429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.190:33671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.190
--webserver_port=46841
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:53:34.879936 1429 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:53:34.882067 1429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260204 02:53:34.891136 1491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:34.891072 1490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:34.892323 1493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:53:34.893497 1429 server_base.cc:1060] running on GCE node
I20260204 02:53:34.894275 1429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:53:34.895946 1429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:53:34.897249 1429 hybrid_clock.cc:648] HybridClock initialized: now 1770173614897201 us; error 30 us; skew 500 ppm
I20260204 02:53:34.897711 1429 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:53:34.900202 1429 webserver.cc:492] Webserver started at http://127.0.104.190:46841/ using document root <none> and password file <none>
I20260204 02:53:34.901075 1429 fs_manager.cc:362] Metadata directory not provided
I20260204 02:53:34.901201 1429 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:53:34.908582 1429 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260204 02:53:34.912016 1499 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:34.913594 1429 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260204 02:53:34.913749 1429 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/wal
uuid: "3e85e1dc6a5d4568ad365ba1d77398ff"
format_stamp: "Formatted at 2026-02-04 02:53:30 on dist-test-slave-xxhz"
I20260204 02:53:34.914583 1429 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:53:34.945418 1429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:53:34.946422 1429 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:53:34.946736 1429 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260204 02:53:34.950148 1349 connection.cc:537] client connection to 127.0.104.189:46765 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:53:34.950829 1349 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.189:46765)
W20260204 02:53:34.951335 1349 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.189:46765)
I20260204 02:53:34.951375 1433 consensus_queue.cc:1048] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260204 02:53:34.952327 1349 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.189:46765)
W20260204 02:53:34.953294 1349 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.189:46765)
I20260204 02:53:34.965020 1433 consensus_queue.cc:1048] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:34.986753 1429 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.190:33671
I20260204 02:53:34.986785 1558 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.190:33671 every 8 connection(s)
I20260204 02:53:34.988667 1429 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-0/data/info.pb
I20260204 02:53:34.989379 1412 raft_consensus.cc:493] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:34.989589 1412 raft_consensus.cc:515] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.990200 1412 leader_election.cc:290] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:34.990705 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b883ef926504d489f3a1c0aec6db438" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:34.990911 1019 raft_consensus.cc:2468] T 2b883ef926504d489f3a1c0aec6db438 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 0.
I20260204 02:53:34.991109 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b883ef926504d489f3a1c0aec6db438" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5" is_pre_election: true
I20260204 02:53:34.991339 1091 leader_election.cc:304] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:34.991374 1294 raft_consensus.cc:2468] T 2b883ef926504d489f3a1c0aec6db438 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 0.
I20260204 02:53:34.991657 1412 raft_consensus.cc:2804] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:34.991740 1412 raft_consensus.cc:493] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:34.991838 1412 raft_consensus.cc:3060] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.992254 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 1429
I20260204 02:53:34.994246 1412 raft_consensus.cc:515] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:34.995112 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b883ef926504d489f3a1c0aec6db438" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
I20260204 02:53:34.995272 1019 raft_consensus.cc:3060] T 2b883ef926504d489f3a1c0aec6db438 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:34.997781 1559 sys_catalog.cc:263] Verifying existing consensus state
I20260204 02:53:34.997808 1019 raft_consensus.cc:2468] T 2b883ef926504d489f3a1c0aec6db438 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
I20260204 02:53:34.998194 1091 leader_election.cc:304] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:34.998463 1464 raft_consensus.cc:2804] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:35.000138 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b883ef926504d489f3a1c0aec6db438" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5"
I20260204 02:53:35.000329 1294 raft_consensus.cc:3060] T 2b883ef926504d489f3a1c0aec6db438 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:35.002179 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: Bootstrap starting.
I20260204 02:53:35.002707 1294 raft_consensus.cc:2468] T 2b883ef926504d489f3a1c0aec6db438 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
I20260204 02:53:35.003296 1412 leader_election.cc:290] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 election: Requested vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:35.011278 1412 raft_consensus.cc:493] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:35.011556 1412 raft_consensus.cc:515] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:35.012732 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f0852d9159a45a69935c76182584be3" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:35.013042 1019 raft_consensus.cc:2468] T 6f0852d9159a45a69935c76182584be3 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 0.
I20260204 02:53:35.013346 1559 log.cc:826] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: Log is configured to *not* fsync() on all Append() calls
I20260204 02:53:35.013535 1091 leader_election.cc:304] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:35.014513 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f0852d9159a45a69935c76182584be3" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5" is_pre_election: true
I20260204 02:53:35.014766 1294 raft_consensus.cc:2468] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 0.
I20260204 02:53:35.014778 1464 raft_consensus.cc:697] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Running, Role: LEADER
I20260204 02:53:35.015275 1412 leader_election.cc:290] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:35.014549 1563 raft_consensus.cc:2804] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:35.015484 1563 raft_consensus.cc:493] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:35.015647 1563 raft_consensus.cc:3060] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:35.019174 1563 raft_consensus.cc:515] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:35.020309 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f0852d9159a45a69935c76182584be3" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
I20260204 02:53:35.020569 1019 raft_consensus.cc:3060] T 6f0852d9159a45a69935c76182584be3 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:35.023733 1019 raft_consensus.cc:2468] T 6f0852d9159a45a69935c76182584be3 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
I20260204 02:53:35.024191 1091 leader_election.cc:304] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:35.024577 1412 raft_consensus.cc:2804] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:35.015153 1464 consensus_queue.cc:237] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:35.036872 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.190:33671
I20260204 02:53:35.037340 1563 leader_election.cc:290] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 election: Requested vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:35.037765 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f0852d9159a45a69935c76182584be3" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5"
I20260204 02:53:35.037977 1294 raft_consensus.cc:3060] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:35.039172 1412 raft_consensus.cc:697] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Running, Role: LEADER
I20260204 02:53:35.039485 1412 consensus_queue.cc:237] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:35.040380 1294 raft_consensus.cc:2468] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
I20260204 02:53:35.044767 1204 heartbeater.cc:344] Connected to a master server at 127.0.104.190:33671
I20260204 02:53:35.046509 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260204 02:53:35.047201 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: Bootstrap complete.
I20260204 02:53:35.052760 1559 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:35.053364 1559 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e85e1dc6a5d4568ad365ba1d77398ff, State: Initialized, Role: FOLLOWER
I20260204 02:53:35.053952 1559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:35.054652 1569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:35.054929 1569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:35.055804 1559 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: configured and running, proceeding with master startup.
I20260204 02:53:35.061874 1417 raft_consensus.cc:493] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:35.062057 1417 raft_consensus.cc:515] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:35.062700 1417 leader_election.cc:290] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:35.063375 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bbcb21479124152831cf7d022fa4e5f" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8" is_pre_election: true
I20260204 02:53:35.063609 1158 raft_consensus.cc:2468] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 0.
I20260204 02:53:35.063742 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bbcb21479124152831cf7d022fa4e5f" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:35.064023 1019 raft_consensus.cc:2468] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 0.
I20260204 02:53:35.064102 1227 leader_election.cc:304] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:35.064355 1417 raft_consensus.cc:2804] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:35.064447 1417 raft_consensus.cc:493] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:35.064543 1417 raft_consensus.cc:3060] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:35.066555 1417 raft_consensus.cc:515] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:35.067133 1417 leader_election.cc:290] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 election: Requested vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:35.067930 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bbcb21479124152831cf7d022fa4e5f" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
I20260204 02:53:35.068095 1019 raft_consensus.cc:3060] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:35.068107 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bbcb21479124152831cf7d022fa4e5f" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8"
I20260204 02:53:35.068234 1158 raft_consensus.cc:3060] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:35.070626 1019 raft_consensus.cc:2468] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 1.
I20260204 02:53:35.071028 1227 leader_election.cc:304] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:35.071251 1417 raft_consensus.cc:2804] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:35.071238 1158 raft_consensus.cc:2468] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 1.
I20260204 02:53:35.073212 1417 raft_consensus.cc:697] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [term 1 LEADER]: Becoming Leader. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Running, Role: LEADER
I20260204 02:53:35.073558 1417 consensus_queue.cc:237] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:35.084048 1583 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:35.084194 1583 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: loading cluster ID for follower catalog manager: success
I20260204 02:53:35.087461 1340 heartbeater.cc:344] Connected to a master server at 127.0.104.190:33671
I20260204 02:53:35.087670 1340 heartbeater.cc:499] Master 127.0.104.190:33671 was elected leader, sending a full tablet report...
I20260204 02:53:35.089993 1583 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: acquiring CA information for follower catalog manager: success
I20260204 02:53:35.094092 1524 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" instance_seqno: 1770173613100203) as {username='slave'} at 127.0.104.131:39637; Asking this server to re-register.
I20260204 02:53:35.094797 1340 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:35.096993 1524 ts_manager.cc:194] Registered new tserver with Master: e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:35.100323 1583 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260204 02:53:35.100690 1429 master_runner.cc:186] Error getting master registration for 127.0.104.189:46765: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111),
I20260204 02:53:35.103876 1429 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:53:35.165725 1588 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:53:35.165907 1588 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:35.172643 1588 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:53:35.181001 817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:35.190361 1534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202e4a0debb24c8aa89a4724d72fcf37" candidate_term: 3 candidate_status { last_received { term: 2 index: 34 } } ignore_live_leader: false dest_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" is_pre_election: true
I20260204 02:53:35.191042 1534 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 202e4a0debb24c8aa89a4724d72fcf37 in term 1.
W20260204 02:53:35.191291 817 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:35.191672 816 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff; no voters: 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:35.191952 1588 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260204 02:53:35.192082 1588 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:53:35.192153 1588 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 2 FOLLOWER]: Advancing to term 3
I20260204 02:53:35.194000 1588 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:35.194510 1588 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 3 election: Requested vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
I20260204 02:53:35.194911 1534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202e4a0debb24c8aa89a4724d72fcf37" candidate_term: 3 candidate_status { last_received { term: 2 index: 34 } } ignore_live_leader: false dest_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff"
I20260204 02:53:35.195109 1534 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 1 FOLLOWER]: Advancing to term 3
W20260204 02:53:35.196280 817 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:35.200220 1534 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 202e4a0debb24c8aa89a4724d72fcf37 in term 3.
I20260204 02:53:35.200605 816 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff; no voters: 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:35.200985 1588 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 3 FOLLOWER]: Leader election won for term 3
I20260204 02:53:35.202055 1588 raft_consensus.cc:697] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 3 LEADER]: Becoming Leader. State: Replica: 202e4a0debb24c8aa89a4724d72fcf37, State: Running, Role: LEADER
I20260204 02:53:35.202445 1588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 34, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:35.203881 1592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 202e4a0debb24c8aa89a4724d72fcf37. Latest consensus state: current_term: 3 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:35.204190 1592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: LEADER
I20260204 02:53:35.204818 1593 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260204 02:53:35.207115 1593 catalog_manager.cc:679] Loaded metadata for table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]
I20260204 02:53:35.207618 1593 catalog_manager.cc:679] Loaded metadata for table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]
I20260204 02:53:35.207998 1593 catalog_manager.cc:679] Loaded metadata for table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]
I20260204 02:53:35.208344 1593 catalog_manager.cc:679] Loaded metadata for table default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796]
I20260204 02:53:35.208684 1593 catalog_manager.cc:679] Loaded metadata for table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]
I20260204 02:53:35.209017 1593 catalog_manager.cc:679] Loaded metadata for table default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2]
I20260204 02:53:35.209312 1593 catalog_manager.cc:679] Loaded metadata for table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29]
I20260204 02:53:35.209632 1593 catalog_manager.cc:679] Loaded metadata for table default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5]
I20260204 02:53:35.209942 1593 catalog_manager.cc:679] Loaded metadata for table default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f]
I20260204 02:53:35.211393 1593 tablet_loader.cc:96] loaded metadata for tablet 2b883ef926504d489f3a1c0aec6db438 (table default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5])
I20260204 02:53:35.211920 1593 tablet_loader.cc:96] loaded metadata for tablet 6f0852d9159a45a69935c76182584be3 (table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b])
I20260204 02:53:35.212378 1593 tablet_loader.cc:96] loaded metadata for tablet 8bbcb21479124152831cf7d022fa4e5f (table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793])
I20260204 02:53:35.212651 1593 tablet_loader.cc:96] loaded metadata for tablet 8c77d21f66a640d88bf1865d6176c969 (table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29])
I20260204 02:53:35.212944 1593 tablet_loader.cc:96] loaded metadata for tablet 8f1bc638fe8e48189c15d18209efdd1d (table default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796])
I20260204 02:53:35.213228 1593 tablet_loader.cc:96] loaded metadata for tablet 9a80cbfc27b54a5d945683c39020a027 (table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76])
I20260204 02:53:35.213519 1593 tablet_loader.cc:96] loaded metadata for tablet b321632c62d545d9b3ce83683af6de42 (table default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f])
I20260204 02:53:35.213806 1593 tablet_loader.cc:96] loaded metadata for tablet b8094eec37cd40b9b1f4931f24d824a8 (table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2])
I20260204 02:53:35.214219 1593 tablet_loader.cc:96] loaded metadata for tablet f66e2750d8c343c0bfa567fd36bc4da0 (table default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2])
I20260204 02:53:35.214435 1593 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260204 02:53:35.215245 1593 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:35.215305 1593 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260204 02:53:35.216507 1593 catalog_manager.cc:1514] Loading token signing keys...
I20260204 02:53:35.217370 1593 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Loaded TSK: 0
I20260204 02:53:35.218178 1593 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260204 02:53:35.218945 1593 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260204 02:53:35.219789 1593 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260204 02:53:35.228945 830 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38944:
name: "default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:35.229651 829 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38944:
name: "default.Table_c26f0462e2664a018e18bf0ef613c017"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:35.230268 822 master_service.cc:954] ReplaceTablet: received request to replace tablet 8c77d21f66a640d88bf1865d6176c969 from {username='slave'} at 127.0.0.1:38944
I20260204 02:53:35.230374 825 master_service.cc:954] ReplaceTablet: received request to replace tablet 9a80cbfc27b54a5d945683c39020a027 from {username='slave'} at 127.0.0.1:38944
I20260204 02:53:35.230759 823 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38944:
table { table_name: "default.Table_69c350f50a63410c9da5a5e7b0862cf6" } modify_external_catalogs: true
I20260204 02:53:35.231040 823 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38944:
table { table_name: "default.Table_69c350f50a63410c9da5a5e7b0862cf6" } modify_external_catalogs: true
I20260204 02:53:35.231431 821 master_service.cc:954] ReplaceTablet: received request to replace tablet 6f0852d9159a45a69935c76182584be3 from {username='slave'} at 127.0.0.1:38944
I20260204 02:53:35.231379 828 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38944:
name: "default.Table_e7eb44e87e2543d69bd03454b10c8310"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:35.232026 826 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38944:
name: "default.Table_6bba790402724856976fb9e10e3c48fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:35.232484 824 master_service.cc:954] ReplaceTablet: received request to replace tablet f66e2750d8c343c0bfa567fd36bc4da0 from {username='slave'} at 127.0.0.1:38944
I20260204 02:53:35.233897 827 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38944:
table { table_name: "default.Table_a16e01078d0342e7abbb4ec634032416" } modify_external_catalogs: true
I20260204 02:53:35.234150 827 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38944:
table { table_name: "default.Table_a16e01078d0342e7abbb4ec634032416" } modify_external_catalogs: true
I20260204 02:53:35.235581 1433 raft_consensus.cc:493] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:35.235754 1433 raft_consensus.cc:515] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:35.236424 1433 leader_election.cc:290] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:53:35.236819 817 consensus_peers.cc:597] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 -> Peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Couldn't send request to peer 78eb37be156448e8879b0df55832ab2d. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260204 02:53:35.237476 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f1bc638fe8e48189c15d18209efdd1d" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8" is_pre_election: true
I20260204 02:53:35.237700 1158 raft_consensus.cc:2468] T 8f1bc638fe8e48189c15d18209efdd1d P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 0.
I20260204 02:53:35.238099 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f1bc638fe8e48189c15d18209efdd1d" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5" is_pre_election: true
I20260204 02:53:35.238224 1534 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Refusing update from remote peer 202e4a0debb24c8aa89a4724d72fcf37: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 36. (index mismatch)
I20260204 02:53:35.238297 1294 raft_consensus.cc:2468] T 8f1bc638fe8e48189c15d18209efdd1d P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 0.
I20260204 02:53:35.238751 954 leader_election.cc:304] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:35.239007 1592 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 4, Time since last communication: 0.000s
I20260204 02:53:35.239362 1433 raft_consensus.cc:2804] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:35.239446 1433 raft_consensus.cc:493] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:35.239569 1433 raft_consensus.cc:3060] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
F20260204 02:53:35.241079 886 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_3a403dbfa3044e55abc223b3acd75b6b, message={"server":"","servicePrincipal":"","db":"default","table":"table_3a403dbfa3044e55abc223b3acd75b6b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_3a403dbfa3044e55abc223b3acd75b6b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_3a403dbfa3044e55abc223b3acd75b6b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_3a403dbfa3044e55abc223b3acd75b6b\",\"kudu.table_id\":\"333921075666492fb4e06fa35e04487b\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_0312705ce5e140d4bcf88a92cab2bb7b, message={"server":"","servicePrincipal":"","db":"default","table":"table_0312705ce5e140d4bcf88a92cab2bb7b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0312705ce5e140d4bcf88a92cab2bb7b\",\"kudu.table_id\":\"8df7ca3e2276436e8bdb4480fc3dd711\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f5c03d2ddcd google::LogMessage::Fail() at ??:0
@ 0x7f5c03d31b93 google::LogMessage::SendToLog() at ??:0
I20260204 02:53:35.245919 1433 raft_consensus.cc:515] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
@ 0x7f5c03d2d7cc google::LogMessage::Flush() at ??:0
I20260204 02:53:35.247136 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f1bc638fe8e48189c15d18209efdd1d" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5"
I20260204 02:53:35.247327 1294 raft_consensus.cc:3060] T 8f1bc638fe8e48189c15d18209efdd1d P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:35.249794 1294 raft_consensus.cc:2468] T 8f1bc638fe8e48189c15d18209efdd1d P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 1.
I20260204 02:53:35.250682 954 leader_election.cc:304] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:35.250996 1455 raft_consensus.cc:2804] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:35.251119 1455 raft_consensus.cc:697] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Running, Role: LEADER
I20260204 02:53:35.251397 1455 consensus_queue.cc:237] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
@ 0x7f5c03d2da11 google::LogMessage::~LogMessage() at ??:0
I20260204 02:53:35.256538 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f1bc638fe8e48189c15d18209efdd1d" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8"
I20260204 02:53:35.256747 1158 raft_consensus.cc:3060] T 8f1bc638fe8e48189c15d18209efdd1d P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:35.259670 1158 raft_consensus.cc:2468] T 8f1bc638fe8e48189c15d18209efdd1d P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 1.
@ 0x7f5c1a7617f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260204 02:53:35.262936 1065 heartbeater.cc:344] Connected to a master server at 127.0.104.190:33671
I20260204 02:53:35.263180 1433 leader_election.cc:290] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 election: Requested vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
@ 0x7f5c1a7600a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f5c1a767d1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f5c1a767b99 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f5c06aa2f99 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5c0a0af6db start_thread at ??:0
@ 0x7f5c02bde71f clone at ??:0
I20260204 02:53:35.303455 1524 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" instance_seqno: 1770173612117836) as {username='slave'} at 127.0.104.129:51191; Asking this server to re-register.
I20260204 02:53:35.303758 1602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:35.304041 1602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:35.304277 1569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 202e4a0debb24c8aa89a4724d72fcf37. Latest consensus state: current_term: 3 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:35.304445 1569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:35.304442 1065 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:35.306283 1524 ts_manager.cc:194] Registered new tserver with Master: 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:35.334247 1595 mvcc.cc:204] Tried to move back new op lower bound from 7250631125644673024 to 7250631120020959232. Current Snapshot: MvccSnapshot[applied={T|T < 7250631125644673024 or (T in {7250631125644673024})}]
I20260204 02:53:35.368605 1455 consensus_queue.cc:1048] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:35.376462 1524 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" instance_seqno: 1770173612604702) as {username='slave'} at 127.0.104.130:37673; Asking this server to re-register.
I20260204 02:53:35.378715 1433 consensus_queue.cc:1048] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:35.379320 1204 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:35.381417 1524 ts_manager.cc:194] Registered new tserver with Master: 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:53:35.471221 1227 connection.cc:537] client connection to 127.0.104.188:38789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:53:35.472075 1500 connection.cc:537] server connection from 127.0.0.1:59882 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:53:35.472878 1091 connection.cc:537] client connection to 127.0.104.188:38789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260204 02:53:35.474486 1433 consensus_queue.cc:1048] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:35.481356 1416 consensus_queue.cc:1048] T b321632c62d545d9b3ce83683af6de42 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:35.492568 1412 consensus_queue.cc:1048] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:35.498170 1563 consensus_queue.cc:1048] T b8094eec37cd40b9b1f4931f24d824a8 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:35.616219 1417 consensus_queue.cc:1048] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:35.624938 1622 consensus_queue.cc:1048] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:35.742544 1569 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 202e4a0debb24c8aa89a4724d72fcf37)
I20260204 02:53:35.742740 1569 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:35.744273 1569 leader_election.cc:290] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:35.745198 1500 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
W20260204 02:53:35.746398 1503 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:53:35.746627 1500 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:35.746744 1500 leader_election.cc:304] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e85e1dc6a5d4568ad365ba1d77398ff; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:35.747009 1569 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260204 02:53:35.877952 1623 consensus_queue.cc:1048] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:35.884191 1417 consensus_queue.cc:1048] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:35.887239 1563 consensus_queue.cc:1048] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:35.893375 1412 consensus_queue.cc:1048] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:36.013077 1412 consensus_queue.cc:1048] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:36.017663 1563 consensus_queue.cc:1048] T 2b883ef926504d489f3a1c0aec6db438 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:36.173789 1455 consensus_queue.cc:1048] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:36.180114 1416 consensus_queue.cc:1048] T 8f1bc638fe8e48189c15d18209efdd1d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:36.425932 1638 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 202e4a0debb24c8aa89a4724d72fcf37)
I20260204 02:53:36.426116 1638 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:36.426754 1638 leader_election.cc:290] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:36.428469 1503 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:53:36.428646 1500 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:36.428754 1500 leader_election.cc:304] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e85e1dc6a5d4568ad365ba1d77398ff; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:36.428946 1638 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260204 02:53:37.380182 1653 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 202e4a0debb24c8aa89a4724d72fcf37)
I20260204 02:53:37.380386 1653 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:37.381024 1653 leader_election.cc:290] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:37.382856 1503 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:53:37.383030 1500 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:37.383149 1500 leader_election.cc:304] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e85e1dc6a5d4568ad365ba1d77398ff; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:37.383419 1653 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260204 02:53:38.276281 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.188:38789
--webserver_interface=127.0.104.188
--webserver_port=40055
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260204 02:53:38.370445 1672 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 202e4a0debb24c8aa89a4724d72fcf37)
I20260204 02:53:38.370687 1672 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:38.371295 1672 leader_election.cc:290] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:38.373129 1503 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:53:38.373831 1500 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:38.373948 1500 leader_election.cc:304] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e85e1dc6a5d4568ad365ba1d77398ff; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:38.374157 1672 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260204 02:53:38.588645 1666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:53:38.588941 1666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:53:38.589107 1666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260204 02:53:38.596727 1666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260204 02:53:38.596812 1666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:53:38.596904 1666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260204 02:53:38.596997 1666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260204 02:53:38.606015 1666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.188:38789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.188
--webserver_port=40055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:53:38.607656 1666 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:53:38.609323 1666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260204 02:53:38.616899 1680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:38.617595 1681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:38.618764 1683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:53:38.619105 1666 server_base.cc:1060] running on GCE node
I20260204 02:53:38.619849 1666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:53:38.621258 1666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:53:38.622496 1666 hybrid_clock.cc:648] HybridClock initialized: now 1770173618622447 us; error 39 us; skew 500 ppm
I20260204 02:53:38.622843 1666 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:53:38.624956 1666 webserver.cc:492] Webserver started at http://127.0.104.188:40055/ using document root <none> and password file <none>
I20260204 02:53:38.625646 1666 fs_manager.cc:362] Metadata directory not provided
I20260204 02:53:38.625782 1666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:53:38.630748 1666 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260204 02:53:38.633361 1689 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:38.634907 1666 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260204 02:53:38.635072 1666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
uuid: "202e4a0debb24c8aa89a4724d72fcf37"
format_stamp: "Formatted at 2026-02-04 02:53:31 on dist-test-slave-xxhz"
I20260204 02:53:38.635783 1666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:53:38.650475 1666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:53:38.651225 1666 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:53:38.651556 1666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260204 02:53:38.671182 1666 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.188:38789
I20260204 02:53:38.671203 1740 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.188:38789 every 8 connection(s)
I20260204 02:53:38.672768 1666 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
I20260204 02:53:38.673066 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 1666
I20260204 02:53:38.680315 1741 sys_catalog.cc:263] Verifying existing consensus state
I20260204 02:53:38.683561 1741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap starting.
I20260204 02:53:38.696152 1741 log.cc:826] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Log is configured to *not* fsync() on all Append() calls
I20260204 02:53:38.726099 1741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap replayed 1/1 log segments. Stats: ops{read=39 overwritten=0 applied=34 ignored=0} inserts{seen=24 ignored=0} mutations{seen=20 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20260204 02:53:38.726787 1741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap complete.
I20260204 02:53:38.731621 1741 raft_consensus.cc:359] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 3 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:38.734709 1741 raft_consensus.cc:740] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202e4a0debb24c8aa89a4724d72fcf37, State: Initialized, Role: FOLLOWER
I20260204 02:53:38.735509 1741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 3.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:38.736354 1746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:38.736570 1746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:38.737401 1741 sys_catalog.cc:565] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: configured and running, proceeding with master startup.
I20260204 02:53:38.751327 1760 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:38.751442 1760 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: loading cluster ID for follower catalog manager: success
I20260204 02:53:38.765138 1760 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: acquiring CA information for follower catalog manager: success
I20260204 02:53:38.769428 1760 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260204 02:53:38.781529 1666 master_runner.cc:186] Error getting master registration for 127.0.104.189:46765: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111),
I20260204 02:53:38.787251 1666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:53:38.795622 1203 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:38.798935 1706 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" instance_seqno: 1770173612604702) as {username='slave'} at 127.0.104.130:46309; Asking this server to re-register.
I20260204 02:53:38.799741 1203 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:38.801261 1706 ts_manager.cc:194] Registered new tserver with Master: 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:38.860821 1064 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:38.862715 1706 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" instance_seqno: 1770173612117836) as {username='slave'} at 127.0.104.129:57687; Asking this server to re-register.
I20260204 02:53:38.863327 1064 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:38.864652 1706 ts_manager.cc:194] Registered new tserver with Master: 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:38.877341 1339 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:38.879151 1706 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" instance_seqno: 1770173613100203) as {username='slave'} at 127.0.104.131:45801; Asking this server to re-register.
I20260204 02:53:38.879808 1339 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:38.881217 1706 ts_manager.cc:194] Registered new tserver with Master: e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:39.199129 1746 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:39.199420 1746 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:39.201287 1746 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
I20260204 02:53:39.201876 1534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202e4a0debb24c8aa89a4724d72fcf37" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: false dest_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" is_pre_election: true
W20260204 02:53:39.201885 1693 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:39.202090 1534 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 202e4a0debb24c8aa89a4724d72fcf37 in term 3.
I20260204 02:53:39.202747 1692 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff; no voters:
I20260204 02:53:39.203069 1746 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260204 02:53:39.203220 1746 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:39.203346 1746 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 3 FOLLOWER]: Advancing to term 4
W20260204 02:53:39.203390 1693 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:39.206929 1746 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:39.207724 1746 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 4 election: Requested vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
I20260204 02:53:39.207991 1534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202e4a0debb24c8aa89a4724d72fcf37" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: false dest_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff"
I20260204 02:53:39.208163 1534 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 3 FOLLOWER]: Advancing to term 4
W20260204 02:53:39.208827 1693 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:39.209998 1534 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 202e4a0debb24c8aa89a4724d72fcf37 in term 4.
I20260204 02:53:39.210404 1692 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff; no voters: 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:39.210757 1746 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 4 FOLLOWER]: Leader election won for term 4
I20260204 02:53:39.211223 1746 raft_consensus.cc:697] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 4 LEADER]: Becoming Leader. State: Replica: 202e4a0debb24c8aa89a4724d72fcf37, State: Running, Role: LEADER
I20260204 02:53:39.211707 1746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 3.39, Last appended by leader: 39, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:39.213593 1772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 202e4a0debb24c8aa89a4724d72fcf37. Latest consensus state: current_term: 4 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:39.213819 1772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: LEADER
I20260204 02:53:39.621801 1534 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Refusing update from remote peer 202e4a0debb24c8aa89a4724d72fcf37: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 4 index: 40. (index mismatch)
I20260204 02:53:39.622581 1772 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 34, Time since last communication: 0.000s
I20260204 02:53:39.626017 1775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 202e4a0debb24c8aa89a4724d72fcf37. Latest consensus state: current_term: 4 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:39.626441 1775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:39.632683 1746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:39.632927 1746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: LEADER
I20260204 02:53:39.633106 1746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:39.633296 1746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: LEADER
W20260204 02:53:39.634295 1693 consensus_peers.cc:597] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 -> Peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Couldn't send request to peer 78eb37be156448e8879b0df55832ab2d. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260204 02:53:39.637815 1786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:39.637915 1775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:39.638087 1786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:39.638190 1775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:39.733722 1774 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260204 02:53:39.736205 1774 catalog_manager.cc:679] Loaded metadata for table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]
I20260204 02:53:39.736763 1774 catalog_manager.cc:679] Loaded metadata for table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]
I20260204 02:53:39.737071 1774 catalog_manager.cc:679] Loaded metadata for table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]
I20260204 02:53:39.737370 1774 catalog_manager.cc:679] Loaded metadata for table default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796]
I20260204 02:53:39.737639 1774 catalog_manager.cc:679] Loaded metadata for table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]
I20260204 02:53:39.737927 1774 catalog_manager.cc:679] Loaded metadata for table default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2]
I20260204 02:53:39.738178 1774 catalog_manager.cc:679] Loaded metadata for table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29]
I20260204 02:53:39.738569 1774 catalog_manager.cc:679] Loaded metadata for table default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5]
I20260204 02:53:39.738879 1774 catalog_manager.cc:679] Loaded metadata for table default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f]
I20260204 02:53:39.740375 1774 tablet_loader.cc:96] loaded metadata for tablet 01792fc91bba4ba891812bd25e006268 (table default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2])
I20260204 02:53:39.740685 1774 tablet_loader.cc:96] loaded metadata for tablet 15b2b6a7198c4ebb98daf4de751911a6 (table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b])
I20260204 02:53:39.740974 1774 tablet_loader.cc:96] loaded metadata for tablet 1d7efc8694aa49e4ba8ab79721373607 (table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76])
I20260204 02:53:39.741240 1774 tablet_loader.cc:96] loaded metadata for tablet 2b883ef926504d489f3a1c0aec6db438 (table default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5])
I20260204 02:53:39.741472 1774 tablet_loader.cc:96] loaded metadata for tablet 3758b5156dd04de984cd9313459d7cfb (table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29])
I20260204 02:53:39.742202 1774 tablet_loader.cc:96] loaded metadata for tablet 8bbcb21479124152831cf7d022fa4e5f (table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793])
I20260204 02:53:39.742894 1774 tablet_loader.cc:96] loaded metadata for tablet 8f1bc638fe8e48189c15d18209efdd1d (table default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796])
I20260204 02:53:39.743594 1774 tablet_loader.cc:96] loaded metadata for tablet b321632c62d545d9b3ce83683af6de42 (table default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f])
I20260204 02:53:39.743919 1774 tablet_loader.cc:96] loaded metadata for tablet b8094eec37cd40b9b1f4931f24d824a8 (table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2])
I20260204 02:53:39.744769 1774 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260204 02:53:39.745864 1774 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:39.745929 1774 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260204 02:53:39.747205 1774 catalog_manager.cc:1514] Loading token signing keys...
I20260204 02:53:39.748057 1774 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Loaded TSK: 0
I20260204 02:53:39.748833 1774 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260204 02:53:39.749616 1774 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260204 02:53:39.750351 1774 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260204 02:53:39.789477 1274 tablet_service.cc:1505] Processing CreateTablet for tablet 15b2b6a7198c4ebb98daf4de751911a6 (DEFAULT_TABLE table=default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:39.789902 1271 tablet_service.cc:1505] Processing CreateTablet for tablet 01792fc91bba4ba891812bd25e006268 (DEFAULT_TABLE table=default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:39.790146 1274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15b2b6a7198c4ebb98daf4de751911a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:39.790410 1271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01792fc91bba4ba891812bd25e006268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:39.789551 1273 tablet_service.cc:1505] Processing CreateTablet for tablet 3758b5156dd04de984cd9313459d7cfb (DEFAULT_TABLE table=default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:39.791414 1272 tablet_service.cc:1505] Processing CreateTablet for tablet 1d7efc8694aa49e4ba8ab79721373607 (DEFAULT_TABLE table=default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:39.791817 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3758b5156dd04de984cd9313459d7cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:39.799679 1272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7efc8694aa49e4ba8ab79721373607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:39.807065 998 tablet_service.cc:1505] Processing CreateTablet for tablet 3758b5156dd04de984cd9313459d7cfb (DEFAULT_TABLE table=default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:39.807317 1793 tablet_bootstrap.cc:492] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:39.807539 998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3758b5156dd04de984cd9313459d7cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:39.809576 999 tablet_service.cc:1505] Processing CreateTablet for tablet 15b2b6a7198c4ebb98daf4de751911a6 (DEFAULT_TABLE table=default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:39.809718 1793 tablet_bootstrap.cc:654] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:39.809911 999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15b2b6a7198c4ebb98daf4de751911a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:39.811184 996 tablet_service.cc:1505] Processing CreateTablet for tablet 1d7efc8694aa49e4ba8ab79721373607 (DEFAULT_TABLE table=default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:39.811549 996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7efc8694aa49e4ba8ab79721373607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:39.815749 997 tablet_service.cc:1505] Processing CreateTablet for tablet 01792fc91bba4ba891812bd25e006268 (DEFAULT_TABLE table=default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:39.816246 997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01792fc91bba4ba891812bd25e006268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:39.818565 1137 tablet_service.cc:1505] Processing CreateTablet for tablet 3758b5156dd04de984cd9313459d7cfb (DEFAULT_TABLE table=default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:39.819092 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3758b5156dd04de984cd9313459d7cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:39.822060 1795 tablet_bootstrap.cc:492] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:39.819584 1138 tablet_service.cc:1505] Processing CreateTablet for tablet 15b2b6a7198c4ebb98daf4de751911a6 (DEFAULT_TABLE table=default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:39.824035 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15b2b6a7198c4ebb98daf4de751911a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:39.824867 1795 tablet_bootstrap.cc:654] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:39.820152 1135 tablet_service.cc:1505] Processing CreateTablet for tablet 01792fc91bba4ba891812bd25e006268 (DEFAULT_TABLE table=default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:39.829051 1135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01792fc91bba4ba891812bd25e006268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:39.829705 1793 tablet_bootstrap.cc:492] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:39.829869 1793 ts_tablet_manager.cc:1403] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.014s
I20260204 02:53:39.830893 1793 raft_consensus.cc:359] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.831125 1793 raft_consensus.cc:385] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:39.831290 1793 raft_consensus.cc:740] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:39.819832 1136 tablet_service.cc:1505] Processing CreateTablet for tablet 1d7efc8694aa49e4ba8ab79721373607 (DEFAULT_TABLE table=default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:39.831657 1793 consensus_queue.cc:260] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.832028 1136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7efc8694aa49e4ba8ab79721373607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:39.832595 1793 ts_tablet_manager.cc:1434] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260204 02:53:39.832873 1793 tablet_bootstrap.cc:492] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:39.834371 1793 tablet_bootstrap.cc:654] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:39.835933 1339 heartbeater.cc:499] Master 127.0.104.188:38789 was elected leader, sending a full tablet report...
I20260204 02:53:39.847308 1705 catalog_manager.cc:5654] T b8094eec37cd40b9b1f4931f24d824a8 P e2efdafb1246484a98dc5408be6746b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130). New cstate: current_term: 1 leader_uuid: "4dc2992e324049698330803c9ed2e3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } }
I20260204 02:53:39.848516 1705 catalog_manager.cc:5654] T 8f1bc638fe8e48189c15d18209efdd1d P e2efdafb1246484a98dc5408be6746b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129). New cstate: current_term: 1 leader_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } }
I20260204 02:53:39.849020 1705 catalog_manager.cc:5654] T 2b883ef926504d489f3a1c0aec6db438 P e2efdafb1246484a98dc5408be6746b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130). New cstate: current_term: 1 leader_uuid: "4dc2992e324049698330803c9ed2e3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } }
I20260204 02:53:39.849622 1705 catalog_manager.cc:5654] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2efdafb1246484a98dc5408be6746b5 (127.0.104.131). New cstate: current_term: 1 leader_uuid: "e2efdafb1246484a98dc5408be6746b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } health_report { overall_health: HEALTHY } } }
I20260204 02:53:39.851905 1704 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42368:
table { table_name: "default.Table_f734190e49874f9fb2c8dee62aa9e70b" } new_table_name: "default.Table_f9d078d036094cf486755f573c75e8b6" modify_external_catalogs: true
I20260204 02:53:39.861545 1703 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42368:
table { table_name: "default.Table_842bb3fe3b2944498784e63317dd1907" } new_table_name: "default.Table_fb4d5c02e3f443afb1f2f219731403b1" modify_external_catalogs: true
I20260204 02:53:39.862965 1700 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42368:
name: "default.Table_a2c8b05d2d724f50aaf92bddb8a266e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:39.863547 1702 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42368:
table { table_name: "default.Table_3a403dbfa3044e55abc223b3acd75b6b" } new_table_name: "default.Table_7021e56527ac45df985d01556abaeeb8" modify_external_catalogs: true
I20260204 02:53:39.863991 1698 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42368:
name: "default.Table_e971e55de1b44441b34f0913c0d665a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:39.864176 1701 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42368:
name: "default.Table_b896155e926a4679a475f25f74e33530"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:39.864876 1699 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42368:
name: "default.Table_bdc0d01a86b043beb3441b2078794538"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:39.865548 1697 master_service.cc:954] ReplaceTablet: received request to replace tablet 9a80cbfc27b54a5d945683c39020a027 from {username='slave'} at 127.0.0.1:42368
I20260204 02:53:39.868755 1706 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42368:
table { table_name: "default.Table_db7b7860261b41c692b36ca95b2d89e2" } new_table_name: "default.Table_667ea74f761c447c8538d777329b0b9f" modify_external_catalogs: true
I20260204 02:53:39.872291 1697 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42368:
name: "default.Table_6bba790402724856976fb9e10e3c48fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:39.876770 1271 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f0852d9159a45a69935c76182584be3 with delete_type TABLET_DATA_DELETED (replaced by tablet 15b2b6a7198c4ebb98daf4de751911a6) from {username='slave'} at 127.0.0.1:41654
I20260204 02:53:39.880625 1273 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a80cbfc27b54a5d945683c39020a027 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d7efc8694aa49e4ba8ab79721373607) from {username='slave'} at 127.0.0.1:41654
I20260204 02:53:39.881819 1272 tablet_service.cc:1552] Processing DeleteTablet for tablet f66e2750d8c343c0bfa567fd36bc4da0 with delete_type TABLET_DATA_DELETED (replaced by tablet 01792fc91bba4ba891812bd25e006268) from {username='slave'} at 127.0.0.1:41654
I20260204 02:53:39.882103 1273 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c77d21f66a640d88bf1865d6176c969 with delete_type TABLET_DATA_DELETED (replaced by tablet 3758b5156dd04de984cd9313459d7cfb) from {username='slave'} at 127.0.0.1:41654
I20260204 02:53:39.883721 1705 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42368:
table { table_name: "default.Table_a16e01078d0342e7abbb4ec634032416" } modify_external_catalogs: true
I20260204 02:53:39.883980 1705 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42368:
table { table_name: "default.Table_a16e01078d0342e7abbb4ec634032416" } modify_external_catalogs: true
I20260204 02:53:39.884994 1795 tablet_bootstrap.cc:492] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:39.885175 1795 ts_tablet_manager.cc:1403] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.003s
I20260204 02:53:39.886147 1795 raft_consensus.cc:359] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.886415 1795 raft_consensus.cc:385] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:39.886528 1795 raft_consensus.cc:740] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:39.886801 1795 consensus_queue.cc:260] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.887356 1064 heartbeater.cc:499] Master 127.0.104.188:38789 was elected leader, sending a full tablet report...
F20260204 02:53:39.893081 1759 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_3a403dbfa3044e55abc223b3acd75b6b, message={"server":"","servicePrincipal":"","db":"default","table":"table_3a403dbfa3044e55abc223b3acd75b6b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_3a403dbfa3044e55abc223b3acd75b6b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_3a403dbfa3044e55abc223b3acd75b6b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_3a403dbfa3044e55abc223b3acd75b6b\",\"kudu.table_id\":\"333921075666492fb4e06fa35e04487b\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_0312705ce5e140d4bcf88a92cab2bb7b, message={"server":"","servicePrincipal":"","db":"default","table":"table_0312705ce5e140d4bcf88a92cab2bb7b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0312705ce5e140d4bcf88a92cab2bb7b\",\"kudu.table_id\":\"8df7ca3e2276436e8bdb4480fc3dd711\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive);
I20260204 02:53:39.893165 1795 ts_tablet_manager.cc:1434] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
*** Check failure stack trace: ***
@ 0x7f428f0a5dcd google::LogMessage::Fail() at ??:0
I20260204 02:53:39.897559 1795 tablet_bootstrap.cc:492] T 1d7efc8694aa49e4ba8ab79721373607 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
@ 0x7f428f0a9b93 google::LogMessage::SendToLog() at ??:0
I20260204 02:53:39.900132 1795 tablet_bootstrap.cc:654] T 1d7efc8694aa49e4ba8ab79721373607 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
@ 0x7f428f0a57cc google::LogMessage::Flush() at ??:0
I20260204 02:53:39.901651 1797 tablet_bootstrap.cc:492] T 3758b5156dd04de984cd9313459d7cfb P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
@ 0x7f428f0a5a11 google::LogMessage::~LogMessage() at ??:0
I20260204 02:53:39.904268 1797 tablet_bootstrap.cc:654] T 3758b5156dd04de984cd9313459d7cfb P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:39.906369 1793 tablet_bootstrap.cc:492] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:39.906404 1795 tablet_bootstrap.cc:492] T 1d7efc8694aa49e4ba8ab79721373607 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:39.906541 1795 ts_tablet_manager.cc:1403] T 1d7efc8694aa49e4ba8ab79721373607 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260204 02:53:39.906551 1793 ts_tablet_manager.cc:1403] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.074s user 0.032s sys 0.036s
I20260204 02:53:39.907266 1795 raft_consensus.cc:359] T 1d7efc8694aa49e4ba8ab79721373607 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.907274 1793 raft_consensus.cc:359] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.907524 1795 raft_consensus.cc:385] T 1d7efc8694aa49e4ba8ab79721373607 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:39.907527 1793 raft_consensus.cc:385] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:39.907603 1795 raft_consensus.cc:740] T 1d7efc8694aa49e4ba8ab79721373607 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:39.907611 1793 raft_consensus.cc:740] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:39.907868 1793 consensus_queue.cc:260] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.907905 1795 consensus_queue.cc:260] T 1d7efc8694aa49e4ba8ab79721373607 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.908392 1793 ts_tablet_manager.cc:1434] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260204 02:53:39.908716 1793 tablet_bootstrap.cc:492] T 1d7efc8694aa49e4ba8ab79721373607 P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:39.911036 1795 ts_tablet_manager.cc:1434] T 1d7efc8694aa49e4ba8ab79721373607 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260204 02:53:39.912353 1795 tablet_bootstrap.cc:492] T 3758b5156dd04de984cd9313459d7cfb P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:39.912588 1800 tablet_replica.cc:333] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5: stopping tablet replica
I20260204 02:53:39.912779 1800 raft_consensus.cc:2243] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260204 02:53:39.912981 1800 raft_consensus.cc:2272] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:39.913563 1800 ts_tablet_manager.cc:1916] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:39.913823 1795 tablet_bootstrap.cc:654] T 3758b5156dd04de984cd9313459d7cfb P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:39.915534 1793 tablet_bootstrap.cc:654] T 1d7efc8694aa49e4ba8ab79721373607 P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
@ 0x7f42a5ad97f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260204 02:53:39.921633 1800 ts_tablet_manager.cc:1929] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260204 02:53:39.921810 1800 log.cc:1199] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal/wals/6f0852d9159a45a69935c76182584be3
I20260204 02:53:39.922046 1795 tablet_bootstrap.cc:492] T 3758b5156dd04de984cd9313459d7cfb P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:39.922246 1795 ts_tablet_manager.cc:1403] T 3758b5156dd04de984cd9313459d7cfb P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260204 02:53:39.922320 1800 ts_tablet_manager.cc:1950] T 6f0852d9159a45a69935c76182584be3 P e2efdafb1246484a98dc5408be6746b5: Deleting consensus metadata
I20260204 02:53:39.923157 1795 raft_consensus.cc:359] T 3758b5156dd04de984cd9313459d7cfb P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:39.923379 1795 raft_consensus.cc:385] T 3758b5156dd04de984cd9313459d7cfb P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:39.923468 1795 raft_consensus.cc:740] T 3758b5156dd04de984cd9313459d7cfb P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:39.923664 1692 catalog_manager.cc:4985] TS e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): tablet 6f0852d9159a45a69935c76182584be3 (table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]) successfully deleted
@ 0x7f42a5ad80a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260204 02:53:39.925840 1795 consensus_queue.cc:260] T 3758b5156dd04de984cd9313459d7cfb P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:39.927816 1795 ts_tablet_manager.cc:1434] T 3758b5156dd04de984cd9313459d7cfb P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
@ 0x7f42a5adfd1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260204 02:53:39.928736 1797 tablet_bootstrap.cc:492] T 3758b5156dd04de984cd9313459d7cfb P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:39.928805 1800 tablet_replica.cc:333] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5: stopping tablet replica
I20260204 02:53:39.928871 1797 ts_tablet_manager.cc:1403] T 3758b5156dd04de984cd9313459d7cfb P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.006s
I20260204 02:53:39.929029 1800 raft_consensus.cc:2243] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260204 02:53:39.929204 1800 raft_consensus.cc:2272] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:39.929687 1797 raft_consensus.cc:359] T 3758b5156dd04de984cd9313459d7cfb P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:39.929849 1797 raft_consensus.cc:385] T 3758b5156dd04de984cd9313459d7cfb P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:39.929852 1800 ts_tablet_manager.cc:1916] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:39.929924 1797 raft_consensus.cc:740] T 3758b5156dd04de984cd9313459d7cfb P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:39.930181 1797 consensus_queue.cc:260] T 3758b5156dd04de984cd9313459d7cfb P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:39.930588 1203 heartbeater.cc:499] Master 127.0.104.188:38789 was elected leader, sending a full tablet report...
@ 0x7f42a5adfb99 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
I20260204 02:53:39.934486 1797 ts_tablet_manager.cc:1434] T 3758b5156dd04de984cd9313459d7cfb P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260204 02:53:39.938372 1797 tablet_bootstrap.cc:492] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
@ 0x7f4291e1af99 kudu::Thread::SuperviseThread() at ??:0
I20260204 02:53:39.940851 1795 tablet_bootstrap.cc:492] T 01792fc91bba4ba891812bd25e006268 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
@ 0x7f42954276db start_thread at ??:0
@ 0x7f428df5671f clone at ??:0
I20260204 02:53:39.944084 1797 tablet_bootstrap.cc:654] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:39.945966 1795 tablet_bootstrap.cc:654] T 01792fc91bba4ba891812bd25e006268 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:39.946424 1793 tablet_bootstrap.cc:492] T 1d7efc8694aa49e4ba8ab79721373607 P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:39.946583 1793 ts_tablet_manager.cc:1403] T 1d7efc8694aa49e4ba8ab79721373607 P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.016s
I20260204 02:53:39.947542 1793 raft_consensus.cc:359] T 1d7efc8694aa49e4ba8ab79721373607 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.947753 1793 raft_consensus.cc:385] T 1d7efc8694aa49e4ba8ab79721373607 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:39.947846 1793 raft_consensus.cc:740] T 1d7efc8694aa49e4ba8ab79721373607 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:39.948163 1793 consensus_queue.cc:260] T 1d7efc8694aa49e4ba8ab79721373607 P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.948804 1793 ts_tablet_manager.cc:1434] T 1d7efc8694aa49e4ba8ab79721373607 P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260204 02:53:39.949383 1793 tablet_bootstrap.cc:492] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:39.952289 1793 tablet_bootstrap.cc:654] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:39.952739 1648 raft_consensus.cc:493] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:39.952873 1648 raft_consensus.cc:515] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.953593 1648 leader_election.cc:290] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:39.954069 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15b2b6a7198c4ebb98daf4de751911a6" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5" is_pre_election: true
I20260204 02:53:39.954284 1800 ts_tablet_manager.cc:1929] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260204 02:53:39.954439 1800 log.cc:1199] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal/wals/9a80cbfc27b54a5d945683c39020a027
I20260204 02:53:39.954474 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15b2b6a7198c4ebb98daf4de751911a6" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8" is_pre_election: true
I20260204 02:53:39.954869 1800 ts_tablet_manager.cc:1950] T 9a80cbfc27b54a5d945683c39020a027 P e2efdafb1246484a98dc5408be6746b5: Deleting consensus metadata
I20260204 02:53:39.954285 1294 raft_consensus.cc:2468] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 0.
I20260204 02:53:39.955492 954 leader_election.cc:304] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters:
W20260204 02:53:39.955647 952 leader_election.cc:343] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Illegal state: must be running to vote when last-logged opid is not known
I20260204 02:53:39.956321 1797 tablet_bootstrap.cc:492] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:39.956523 1797 ts_tablet_manager.cc:1403] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.003s
I20260204 02:53:39.957340 1648 raft_consensus.cc:2804] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:39.957513 1648 raft_consensus.cc:493] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:39.957580 1648 raft_consensus.cc:3060] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:39.957676 1797 raft_consensus.cc:359] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.957918 1797 raft_consensus.cc:385] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:39.958021 1797 raft_consensus.cc:740] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:39.958925 1797 consensus_queue.cc:260] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.961769 1797 ts_tablet_manager.cc:1434] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260204 02:53:39.962623 1795 tablet_bootstrap.cc:492] T 01792fc91bba4ba891812bd25e006268 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:39.962842 1795 ts_tablet_manager.cc:1403] T 01792fc91bba4ba891812bd25e006268 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
I20260204 02:53:39.962857 1800 tablet_replica.cc:333] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5: stopping tablet replica
I20260204 02:53:39.963038 1800 raft_consensus.cc:2243] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260204 02:53:39.963209 1800 raft_consensus.cc:2272] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:39.963734 1800 ts_tablet_manager.cc:1916] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:39.963804 1795 raft_consensus.cc:359] T 01792fc91bba4ba891812bd25e006268 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.964015 1795 raft_consensus.cc:385] T 01792fc91bba4ba891812bd25e006268 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:39.964089 1795 raft_consensus.cc:740] T 01792fc91bba4ba891812bd25e006268 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:39.964346 1795 consensus_queue.cc:260] T 01792fc91bba4ba891812bd25e006268 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.968812 1648 raft_consensus.cc:515] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.969182 1797 tablet_bootstrap.cc:492] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:39.969492 1648 leader_election.cc:290] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 election: Requested vote from peers 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:39.970849 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15b2b6a7198c4ebb98daf4de751911a6" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8"
I20260204 02:53:39.971233 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15b2b6a7198c4ebb98daf4de751911a6" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5"
I20260204 02:53:39.971413 1294 raft_consensus.cc:3060] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
W20260204 02:53:39.971374 952 leader_election.cc:343] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Illegal state: must be running to vote when last-logged opid is not known
I20260204 02:53:39.972991 1797 tablet_bootstrap.cc:654] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:39.973320 1800 ts_tablet_manager.cc:1929] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260204 02:53:39.973434 1800 log.cc:1199] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal/wals/f66e2750d8c343c0bfa567fd36bc4da0
I20260204 02:53:39.973810 1800 ts_tablet_manager.cc:1950] T f66e2750d8c343c0bfa567fd36bc4da0 P e2efdafb1246484a98dc5408be6746b5: Deleting consensus metadata
I20260204 02:53:39.974054 1294 raft_consensus.cc:2468] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in term 1.
I20260204 02:53:39.974938 1800 tablet_replica.cc:333] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5: stopping tablet replica
I20260204 02:53:39.975118 1800 raft_consensus.cc:2243] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [term 1 LEADER]: Raft consensus shutting down.
I20260204 02:53:39.975612 1800 raft_consensus.cc:2272] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:39.976166 1800 ts_tablet_manager.cc:1916] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:39.976778 954 leader_election.cc:304] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters: 4dc2992e324049698330803c9ed2e3f8
I20260204 02:53:39.977257 1667 raft_consensus.cc:493] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:39.977397 1667 raft_consensus.cc:515] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.978927 1667 leader_election.cc:290] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:39.979529 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7efc8694aa49e4ba8ab79721373607" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:39.979722 1019 raft_consensus.cc:2468] T 1d7efc8694aa49e4ba8ab79721373607 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 0.
I20260204 02:53:39.979996 1795 ts_tablet_manager.cc:1434] T 01792fc91bba4ba891812bd25e006268 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.017s user 0.000s sys 0.003s
I20260204 02:53:39.980193 1091 leader_election.cc:304] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:39.980974 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7efc8694aa49e4ba8ab79721373607" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5" is_pre_election: true
I20260204 02:53:39.981149 1294 raft_consensus.cc:2468] T 1d7efc8694aa49e4ba8ab79721373607 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 0.
I20260204 02:53:39.981995 1797 tablet_bootstrap.cc:492] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:39.981993 1800 ts_tablet_manager.cc:1929] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260204 02:53:39.981998 1793 tablet_bootstrap.cc:492] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:39.982136 1800 log.cc:1199] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal/wals/8c77d21f66a640d88bf1865d6176c969
I20260204 02:53:39.982136 1797 ts_tablet_manager.cc:1403] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20260204 02:53:39.982187 1793 ts_tablet_manager.cc:1403] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.022s
I20260204 02:53:39.982590 1800 ts_tablet_manager.cc:1950] T 8c77d21f66a640d88bf1865d6176c969 P e2efdafb1246484a98dc5408be6746b5: Deleting consensus metadata
I20260204 02:53:39.982961 1797 raft_consensus.cc:359] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.983109 1667 raft_consensus.cc:2804] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:39.983153 1797 raft_consensus.cc:385] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:39.983242 1667 raft_consensus.cc:493] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:39.983322 1667 raft_consensus.cc:3060] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:39.983820 1811 raft_consensus.cc:2804] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:39.983963 1793 raft_consensus.cc:359] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:39.984148 1793 raft_consensus.cc:385] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:39.984213 1793 raft_consensus.cc:740] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:39.984465 1811 raft_consensus.cc:697] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Running, Role: LEADER
I20260204 02:53:39.984445 1793 consensus_queue.cc:260] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:39.984881 1793 ts_tablet_manager.cc:1434] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260204 02:53:39.984761 1811 consensus_queue.cc:237] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.985242 1797 raft_consensus.cc:740] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:39.985500 1797 consensus_queue.cc:260] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.988714 1667 raft_consensus.cc:515] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.989527 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7efc8694aa49e4ba8ab79721373607" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
I20260204 02:53:39.989704 1019 raft_consensus.cc:3060] T 1d7efc8694aa49e4ba8ab79721373607 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:39.992264 1019 raft_consensus.cc:2468] T 1d7efc8694aa49e4ba8ab79721373607 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
I20260204 02:53:39.992710 1091 leader_election.cc:304] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:39.995311 1667 leader_election.cc:290] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 election: Requested vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:39.995438 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7efc8694aa49e4ba8ab79721373607" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5"
I20260204 02:53:39.995682 1294 raft_consensus.cc:3060] T 1d7efc8694aa49e4ba8ab79721373607 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:39.998487 1667 raft_consensus.cc:2804] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:39.998610 1667 raft_consensus.cc:697] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Running, Role: LEADER
I20260204 02:53:39.998922 1667 consensus_queue.cc:237] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:39.999243 1294 raft_consensus.cc:2468] T 1d7efc8694aa49e4ba8ab79721373607 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
I20260204 02:53:40.003922 1797 ts_tablet_manager.cc:1434] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.022s user 0.003s sys 0.008s
I20260204 02:53:40.004403 1797 tablet_bootstrap.cc:492] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:40.006778 1797 tablet_bootstrap.cc:654] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:40.019680 1797 tablet_bootstrap.cc:492] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:40.019843 1797 ts_tablet_manager.cc:1403] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20260204 02:53:40.020660 1797 raft_consensus.cc:359] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:40.020848 1797 raft_consensus.cc:385] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:40.020915 1797 raft_consensus.cc:740] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:40.021136 1797 consensus_queue.cc:260] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:40.021679 1797 ts_tablet_manager.cc:1434] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260204 02:53:40.064126 1815 raft_consensus.cc:493] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:40.064327 1815 raft_consensus.cc:515] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:40.065050 1815 leader_election.cc:290] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:40.065568 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15b2b6a7198c4ebb98daf4de751911a6" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5" is_pre_election: true
I20260204 02:53:40.065618 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15b2b6a7198c4ebb98daf4de751911a6" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:40.065876 1294 raft_consensus.cc:2393] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4dc2992e324049698330803c9ed2e3f8 in current term 1: Already voted for candidate 8d7605b7e0f949b3ba7ffaddd1ca8db3 in this term.
I20260204 02:53:40.066336 1093 leader_election.cc:304] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:53:40.066632 1815 raft_consensus.cc:3060] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:40.069682 1815 raft_consensus.cc:2749] T 15b2b6a7198c4ebb98daf4de751911a6 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260204 02:53:40.106369 1429 master_runner.cc:186] Error getting master registration for 127.0.104.189:46765: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111),
I20260204 02:53:40.106606 1429 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:53:40.165869 1230 connection.cc:537] server connection from 127.0.0.1:41654 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:53:40.167405 1346 connection.cc:537] client connection to 127.0.104.188:38789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:53:40.168247 1346 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.188:38789)
W20260204 02:53:40.168249 1799 negotiation.cc:337] Failed RPC negotiation. Trace:
0204 02:53:40.166862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.188:38789 (local address 127.0.104.129:47913)
0204 02:53:40.167134 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0204 02:53:40.167270 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":160}
W20260204 02:53:40.168558 1346 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.188:38789)
W20260204 02:53:40.168773 1346 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.188:38789)
W20260204 02:53:40.168982 1346 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.188:38789)
I20260204 02:53:40.202641 1657 raft_consensus.cc:493] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:40.202865 1657 raft_consensus.cc:515] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:40.203534 1657 leader_election.cc:290] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:40.203887 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3758b5156dd04de984cd9313459d7cfb" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:40.204102 1019 raft_consensus.cc:2468] T 3758b5156dd04de984cd9313459d7cfb P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 0.
I20260204 02:53:40.204012 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3758b5156dd04de984cd9313459d7cfb" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8" is_pre_election: true
I20260204 02:53:40.204288 1158 raft_consensus.cc:2468] T 3758b5156dd04de984cd9313459d7cfb P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 0.
I20260204 02:53:40.204476 1227 leader_election.cc:304] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:40.204733 1657 raft_consensus.cc:2804] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:40.204833 1657 raft_consensus.cc:493] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:40.204900 1657 raft_consensus.cc:3060] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:40.206903 1657 raft_consensus.cc:515] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:40.207371 1657 leader_election.cc:290] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 election: Requested vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:40.207732 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3758b5156dd04de984cd9313459d7cfb" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
I20260204 02:53:40.207909 1019 raft_consensus.cc:3060] T 3758b5156dd04de984cd9313459d7cfb P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:40.207896 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3758b5156dd04de984cd9313459d7cfb" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8"
I20260204 02:53:40.208073 1158 raft_consensus.cc:3060] T 3758b5156dd04de984cd9313459d7cfb P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:40.209641 1019 raft_consensus.cc:2468] T 3758b5156dd04de984cd9313459d7cfb P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 1.
I20260204 02:53:40.209874 1158 raft_consensus.cc:2468] T 3758b5156dd04de984cd9313459d7cfb P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 1.
I20260204 02:53:40.210038 1227 leader_election.cc:304] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:40.210273 1657 raft_consensus.cc:2804] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:40.210656 1657 raft_consensus.cc:697] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [term 1 LEADER]: Becoming Leader. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Running, Role: LEADER
I20260204 02:53:40.210947 1657 consensus_queue.cc:237] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:40.239323 1815 raft_consensus.cc:493] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:40.239573 1815 raft_consensus.cc:515] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:40.240137 1815 leader_election.cc:290] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:40.240589 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01792fc91bba4ba891812bd25e006268" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5" is_pre_election: true
I20260204 02:53:40.240607 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01792fc91bba4ba891812bd25e006268" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:40.240808 1019 raft_consensus.cc:2468] T 01792fc91bba4ba891812bd25e006268 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 0.
I20260204 02:53:40.240816 1294 raft_consensus.cc:2468] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 0.
I20260204 02:53:40.241241 1091 leader_election.cc:304] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:40.241513 1815 raft_consensus.cc:2804] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:40.241590 1815 raft_consensus.cc:493] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:40.241657 1815 raft_consensus.cc:3060] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:40.243383 1815 raft_consensus.cc:515] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:40.244226 1815 leader_election.cc:290] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 election: Requested vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:40.244449 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01792fc91bba4ba891812bd25e006268" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
I20260204 02:53:40.244475 1294 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01792fc91bba4ba891812bd25e006268" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2efdafb1246484a98dc5408be6746b5"
I20260204 02:53:40.244655 1019 raft_consensus.cc:3060] T 01792fc91bba4ba891812bd25e006268 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:40.244695 1294 raft_consensus.cc:3060] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:40.246712 1019 raft_consensus.cc:2468] T 01792fc91bba4ba891812bd25e006268 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
I20260204 02:53:40.246766 1294 raft_consensus.cc:2468] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
I20260204 02:53:40.247144 1091 leader_election.cc:304] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters:
I20260204 02:53:40.247443 1815 raft_consensus.cc:2804] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:40.247648 1815 raft_consensus.cc:697] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Running, Role: LEADER
I20260204 02:53:40.247886 1815 consensus_queue.cc:237] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
W20260204 02:53:40.263473 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 1: this message will repeat every 5th retry.
I20260204 02:53:40.361945 1833 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 202e4a0debb24c8aa89a4724d72fcf37)
I20260204 02:53:40.362099 1833 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:40.362748 1833 leader_election.cc:290] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:40.364315 1503 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:53:40.364574 1500 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:40.364697 1500 leader_election.cc:304] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e85e1dc6a5d4568ad365ba1d77398ff; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:40.364903 1833 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260204 02:53:40.407545 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 1: this message will repeat every 5th retry.
W20260204 02:53:40.669076 954 consensus_peers.cc:597] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027. This is attempt 1: this message will repeat every 5th retry.
I20260204 02:53:40.765817 1815 consensus_queue.cc:1048] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:40.771221 1844 consensus_queue.cc:1048] T 1d7efc8694aa49e4ba8ab79721373607 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:40.806871 1811 consensus_queue.cc:1048] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:40.812747 1852 consensus_queue.cc:1048] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:40.929040 1854 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 202e4a0debb24c8aa89a4724d72fcf37)
I20260204 02:53:40.929229 1854 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:40.929936 1854 leader_election.cc:290] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:40.930783 1500 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260204 02:53:40.931885 1500 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
W20260204 02:53:40.932035 1503 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:40.932194 1503 leader_election.cc:304] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e85e1dc6a5d4568ad365ba1d77398ff; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:40.932453 1854 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260204 02:53:41.121153 1846 consensus_queue.cc:1048] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:41.127661 1859 consensus_queue.cc:1048] T 3758b5156dd04de984cd9313459d7cfb P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:41.168655 1842 consensus_queue.cc:1048] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:41.173199 1842 consensus_queue.cc:1048] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:41.502708 1865 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 202e4a0debb24c8aa89a4724d72fcf37)
I20260204 02:53:41.502920 1865 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:41.503532 1865 leader_election.cc:290] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:41.505216 1500 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
W20260204 02:53:41.505407 1503 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:41.505546 1503 leader_election.cc:304] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e85e1dc6a5d4568ad365ba1d77398ff; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:41.505798 1865 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260204 02:53:42.093720 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.188:38789
--webserver_interface=127.0.104.188
--webserver_port=40055
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260204 02:53:42.368450 1851 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2efdafb1246484a98dc5408be6746b5)
I20260204 02:53:42.368675 1851 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:42.369457 1851 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:42.370357 1157 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c77d21f66a640d88bf1865d6176c969" candidate_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8" is_pre_election: true
W20260204 02:53:42.370819 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:53:42.371034 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:53:42.371337 1851 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260204 02:53:42.412606 1873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:53:42.412883 1873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:53:42.412945 1873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260204 02:53:42.421182 1873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260204 02:53:42.421267 1873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:53:42.421342 1873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260204 02:53:42.421425 1873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260204 02:53:42.432399 1873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.188:38789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.188
--webserver_port=40055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:53:42.433977 1873 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:53:42.435721 1873 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260204 02:53:42.443634 1882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:42.443614 1881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:53:42.444603 1873 server_base.cc:1060] running on GCE node
W20260204 02:53:42.444792 1884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:53:42.445506 1873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:53:42.446795 1873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:53:42.447973 1873 hybrid_clock.cc:648] HybridClock initialized: now 1770173622447918 us; error 47 us; skew 500 ppm
I20260204 02:53:42.448283 1873 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:53:42.450135 1873 webserver.cc:492] Webserver started at http://127.0.104.188:40055/ using document root <none> and password file <none>
I20260204 02:53:42.450783 1873 fs_manager.cc:362] Metadata directory not provided
I20260204 02:53:42.450892 1873 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:53:42.455766 1873 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260204 02:53:42.458487 1890 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:42.459792 1873 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260204 02:53:42.459930 1873 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
uuid: "202e4a0debb24c8aa89a4724d72fcf37"
format_stamp: "Formatted at 2026-02-04 02:53:31 on dist-test-slave-xxhz"
I20260204 02:53:42.460721 1873 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:53:42.480191 1873 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:53:42.480875 1873 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:53:42.481092 1873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260204 02:53:42.500243 1873 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.188:38789
I20260204 02:53:42.500244 1941 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.188:38789 every 8 connection(s)
I20260204 02:53:42.501610 1873 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
I20260204 02:53:42.507668 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 1873
I20260204 02:53:42.508101 1942 sys_catalog.cc:263] Verifying existing consensus state
I20260204 02:53:42.511693 1942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap starting.
I20260204 02:53:42.523200 1942 log.cc:826] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Log is configured to *not* fsync() on all Append() calls
I20260204 02:53:42.554718 1942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=42 ignored=0} inserts{seen=28 ignored=0} mutations{seen=32 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260204 02:53:42.555260 1942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap complete.
I20260204 02:53:42.561371 1942 raft_consensus.cc:359] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:42.561973 1942 raft_consensus.cc:740] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202e4a0debb24c8aa89a4724d72fcf37, State: Initialized, Role: FOLLOWER
I20260204 02:53:42.562623 1942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:42.563338 1946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:42.563649 1946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:42.564373 1942 sys_catalog.cc:565] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: configured and running, proceeding with master startup.
I20260204 02:53:42.578442 1960 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:42.578531 1960 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: loading cluster ID for follower catalog manager: success
I20260204 02:53:42.587525 1960 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: acquiring CA information for follower catalog manager: success
I20260204 02:53:42.590327 1960 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260204 02:53:42.593793 1873 master_runner.cc:186] Error getting master registration for 127.0.104.189:46765: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111),
I20260204 02:53:42.597524 1873 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:53:42.630947 1203 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:42.631142 1203 heartbeater.cc:499] Master 127.0.104.188:38789 was elected leader, sending a full tablet report...
I20260204 02:53:42.635201 1907 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" instance_seqno: 1770173612604702) as {username='slave'} at 127.0.104.130:41681; Asking this server to re-register.
I20260204 02:53:42.636036 1203 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:42.637810 1907 ts_manager.cc:194] Registered new tserver with Master: 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:42.648571 1339 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:42.650121 1907 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" instance_seqno: 1770173613100203) as {username='slave'} at 127.0.104.131:42305; Asking this server to re-register.
I20260204 02:53:42.650627 1339 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:42.651845 1907 ts_manager.cc:194] Registered new tserver with Master: e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:42.910646 1871 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2efdafb1246484a98dc5408be6746b5)
I20260204 02:53:42.910892 1871 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:42.911715 1871 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:42.912103 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c77d21f66a640d88bf1865d6176c969" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:42.912336 1019 raft_consensus.cc:2468] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 1.
W20260204 02:53:42.912748 1093 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:53:42.912956 1091 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: e2efdafb1246484a98dc5408be6746b5
I20260204 02:53:42.913213 1871 raft_consensus.cc:2804] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260204 02:53:42.913332 1871 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e2efdafb1246484a98dc5408be6746b5)
I20260204 02:53:42.913426 1871 raft_consensus.cc:3060] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Advancing to term 2
I20260204 02:53:42.923851 1871 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:42.924481 1871 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 election: Requested vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:53:42.925698 1093 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:53:42.925722 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c77d21f66a640d88bf1865d6176c969" candidate_uuid: "4dc2992e324049698330803c9ed2e3f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
I20260204 02:53:42.925940 1019 raft_consensus.cc:3060] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Advancing to term 2
I20260204 02:53:42.934412 1064 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:42.934576 1064 heartbeater.cc:499] Master 127.0.104.188:38789 was elected leader, sending a full tablet report...
I20260204 02:53:42.937070 1019 raft_consensus.cc:2468] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc2992e324049698330803c9ed2e3f8 in term 2.
I20260204 02:53:42.937599 1091 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8, 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: e2efdafb1246484a98dc5408be6746b5
I20260204 02:53:42.938050 1871 raft_consensus.cc:2804] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260204 02:53:42.938355 1871 raft_consensus.cc:697] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 2 LEADER]: Becoming Leader. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Running, Role: LEADER
I20260204 02:53:42.938637 1871 consensus_queue.cc:237] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:42.939675 1907 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" instance_seqno: 1770173612117836) as {username='slave'} at 127.0.104.129:58597; Asking this server to re-register.
I20260204 02:53:42.940511 1064 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:42.941807 1907 ts_manager.cc:194] Registered new tserver with Master: 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:43.025564 1969 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 202e4a0debb24c8aa89a4724d72fcf37)
I20260204 02:53:43.025782 1969 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:43.026484 1969 leader_election.cc:290] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:43.028652 1503 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:43.032876 1917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "202e4a0debb24c8aa89a4724d72fcf37" is_pre_election: true
I20260204 02:53:43.033463 1917 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e85e1dc6a5d4568ad365ba1d77398ff in term 4.
I20260204 02:53:43.034458 1500 leader_election.cc:304] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff; no voters: 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:43.035344 1969 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260204 02:53:43.035519 1969 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Starting leader election (detected failure of leader 202e4a0debb24c8aa89a4724d72fcf37)
I20260204 02:53:43.035636 1969 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 4 FOLLOWER]: Advancing to term 5
I20260204 02:53:43.055778 1969 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:43.057005 1969 leader_election.cc:290] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 election: Requested vote from peers 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
I20260204 02:53:43.057578 1917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "202e4a0debb24c8aa89a4724d72fcf37"
I20260204 02:53:43.057775 1917 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 4 FOLLOWER]: Advancing to term 5
W20260204 02:53:43.058092 1503 leader_election.cc:336] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:43.119402 1917 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e85e1dc6a5d4568ad365ba1d77398ff in term 5.
I20260204 02:53:43.120138 1500 leader_election.cc:304] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff; no voters: 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:43.120640 1969 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 5 FOLLOWER]: Leader election won for term 5
I20260204 02:53:43.121063 1969 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [term 5 LEADER]: Becoming Leader. State: Replica: 3e85e1dc6a5d4568ad365ba1d77398ff, State: Running, Role: LEADER
I20260204 02:53:43.121624 1969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:43.123126 1976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e85e1dc6a5d4568ad365ba1d77398ff. Latest consensus state: current_term: 5 leader_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:43.123342 1976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff [sys.catalog]: This master's current role is: LEADER
I20260204 02:53:43.123916 1978 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260204 02:53:43.126528 1978 catalog_manager.cc:679] Loaded metadata for table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]
I20260204 02:53:43.127017 1978 catalog_manager.cc:679] Loaded metadata for table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]
I20260204 02:53:43.127372 1978 catalog_manager.cc:679] Loaded metadata for table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]
I20260204 02:53:43.127748 1978 catalog_manager.cc:679] Loaded metadata for table default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796]
I20260204 02:53:43.128103 1978 catalog_manager.cc:679] Loaded metadata for table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]
I20260204 02:53:43.128441 1978 catalog_manager.cc:679] Loaded metadata for table default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2]
I20260204 02:53:43.128741 1978 catalog_manager.cc:679] Loaded metadata for table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29]
I20260204 02:53:43.129112 1978 catalog_manager.cc:679] Loaded metadata for table default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5]
I20260204 02:53:43.129420 1978 catalog_manager.cc:679] Loaded metadata for table default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f]
I20260204 02:53:43.130970 1978 tablet_loader.cc:96] loaded metadata for tablet 01792fc91bba4ba891812bd25e006268 (table default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2])
I20260204 02:53:43.131366 1978 tablet_loader.cc:96] loaded metadata for tablet 15b2b6a7198c4ebb98daf4de751911a6 (table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b])
I20260204 02:53:43.131721 1978 tablet_loader.cc:96] loaded metadata for tablet 1d7efc8694aa49e4ba8ab79721373607 (table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76])
I20260204 02:53:43.132017 1978 tablet_loader.cc:96] loaded metadata for tablet 2b883ef926504d489f3a1c0aec6db438 (table default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5])
I20260204 02:53:43.132360 1978 tablet_loader.cc:96] loaded metadata for tablet 3758b5156dd04de984cd9313459d7cfb (table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29])
I20260204 02:53:43.133104 1978 tablet_loader.cc:96] loaded metadata for tablet 8bbcb21479124152831cf7d022fa4e5f (table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793])
I20260204 02:53:43.133566 1978 tablet_loader.cc:96] loaded metadata for tablet 8f1bc638fe8e48189c15d18209efdd1d (table default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796])
I20260204 02:53:43.134017 1978 tablet_loader.cc:96] loaded metadata for tablet b321632c62d545d9b3ce83683af6de42 (table default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f])
I20260204 02:53:43.134337 1978 tablet_loader.cc:96] loaded metadata for tablet b8094eec37cd40b9b1f4931f24d824a8 (table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2])
I20260204 02:53:43.134884 1978 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260204 02:53:43.135921 1978 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:43.136018 1978 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260204 02:53:43.137724 1978 catalog_manager.cc:1514] Loading token signing keys...
I20260204 02:53:43.138950 1978 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 3e85e1dc6a5d4568ad365ba1d77398ff: Loaded TSK: 0
I20260204 02:53:43.139797 1978 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260204 02:53:43.140694 1978 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260204 02:53:43.141575 1978 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260204 02:53:43.201828 1523 master_service.cc:954] ReplaceTablet: received request to replace tablet 8c77d21f66a640d88bf1865d6176c969 from {username='slave'} at 127.0.0.1:59840
I20260204 02:53:43.202211 1522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59840:
name: "default.Table_237c40ddb38c410c9ce533b72060712f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:43.203114 1524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59840:
table { table_name: "default.Table_3d3a816d407e429ea458b787f879ede0" } new_table_name: "default.Table_541e82c4f45e4e74926e15734baef2cd" modify_external_catalogs: true
I20260204 02:53:43.203070 1512 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59840:
name: "default.Table_a2c8b05d2d724f50aaf92bddb8a266e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:43.203651 1513 master_service.cc:954] ReplaceTablet: received request to replace tablet 8c77d21f66a640d88bf1865d6176c969 from {username='slave'} at 127.0.0.1:59840
I20260204 02:53:43.203824 1521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59840:
table { table_name: "default.Table_cb3f03c12e314422b462640be08ec293" } modify_external_catalogs: true
I20260204 02:53:43.204413 1521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59840:
table { table_name: "default.Table_cb3f03c12e314422b462640be08ec293" } modify_external_catalogs: true
I20260204 02:53:43.205229 1515 master_service.cc:954] ReplaceTablet: received request to replace tablet 6f0852d9159a45a69935c76182584be3 from {username='slave'} at 127.0.0.1:59840
I20260204 02:53:43.205295 1519 master_service.cc:954] ReplaceTablet: received request to replace tablet f66e2750d8c343c0bfa567fd36bc4da0 from {username='slave'} at 127.0.0.1:59840
I20260204 02:53:43.204630 1518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59840:
name: "default.Table_3f77bbc9c07244dcb50a3b82a1fc8f1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:43.204600 1523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59840:
name: "default.Table_b896155e926a4679a475f25f74e33530"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:43.204551 1516 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59840:
table { table_name: "default.Table_79ae542654da4da5aba4d8c38d6abae7" } modify_external_catalogs: true
I20260204 02:53:43.206169 1515 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59840:
table { table_name: "default.Table_3a403dbfa3044e55abc223b3acd75b6b" } new_table_name: "default.Table_7021e56527ac45df985d01556abaeeb8" modify_external_catalogs: true
I20260204 02:53:43.206347 1513 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59840:
table { table_name: "default.Table_db7b7860261b41c692b36ca95b2d89e2" } new_table_name: "default.Table_667ea74f761c447c8538d777329b0b9f" modify_external_catalogs: true
I20260204 02:53:43.206593 1516 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59840:
table { table_name: "default.Table_79ae542654da4da5aba4d8c38d6abae7" } modify_external_catalogs: true
I20260204 02:53:43.206772 1519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59840:
table { table_name: "default.Table_842bb3fe3b2944498784e63317dd1907" } new_table_name: "default.Table_fb4d5c02e3f443afb1f2f219731403b1" modify_external_catalogs: true
F20260204 02:53:43.209367 1582 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_3a403dbfa3044e55abc223b3acd75b6b, message={"server":"","servicePrincipal":"","db":"default","table":"table_3a403dbfa3044e55abc223b3acd75b6b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_3a403dbfa3044e55abc223b3acd75b6b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_3a403dbfa3044e55abc223b3acd75b6b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_3a403dbfa3044e55abc223b3acd75b6b\",\"kudu.table_id\":\"333921075666492fb4e06fa35e04487b\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_0312705ce5e140d4bcf88a92cab2bb7b, message={"server":"","servicePrincipal":"","db":"default","table":"table_0312705ce5e140d4bcf88a92cab2bb7b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0312705ce5e140d4bcf88a92cab2bb7b\",\"kudu.table_id\":\"8df7ca3e2276436e8bdb4480fc3dd711\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f0e96a6bdcd google::LogMessage::Fail() at ??:0
@ 0x7f0e96a6fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f0e96a6b7cc google::LogMessage::Flush() at ??:0
@ 0x7f0e96a6ba11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f0ead49f7f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f0ead49e0a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f0ead4a5d1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f0ead4a5b99 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f0e997e0f99 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0e9cded6db start_thread at ??:0
@ 0x7f0e9591c71f clone at ??:0
W20260204 02:53:43.398345 1093 connection.cc:537] client connection to 127.0.104.190:33671 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:53:43.398852 1348 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.190:33671)
W20260204 02:53:43.399154 1348 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.190:33671)
W20260204 02:53:43.399323 1348 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.190:33671)
W20260204 02:53:43.400099 1348 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.190:33671)
I20260204 02:53:43.638782 1992 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:43.639010 1992 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:43.640901 1992 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:53:43.641839 1893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:43.643136 1893 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:43.643368 1894 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:43.643522 1894 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:43.643846 1992 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260204 02:53:43.645267 1340 heartbeater.cc:499] Master 127.0.104.190:33671 was elected leader, sending a full tablet report...
W20260204 02:53:44.020859 1093 consensus_peers.cc:597] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969. This is attempt 1: this message will repeat every 5th retry.
I20260204 02:53:44.038094 1019 raft_consensus.cc:1275] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Refusing update from remote peer 4dc2992e324049698330803c9ed2e3f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260204 02:53:44.038703 1990 consensus_queue.cc:1048] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260204 02:53:44.223834 2007 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:44.224050 2007 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:44.224663 2007 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:53:44.226599 1893 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:44.226794 1894 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:44.226951 1894 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:44.227304 2007 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260204 02:53:45.025929 2019 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:45.026167 2019 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:45.026912 2019 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:53:45.028831 1893 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:45.029336 1894 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:53:45.029487 1894 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:53:45.029719 2019 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260204 02:53:45.126943 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.189:46765
--webserver_interface=127.0.104.189
--webserver_port=36571
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260204 02:53:45.451596 2023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:53:45.451923 2023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:53:45.452044 2023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260204 02:53:45.460016 2023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260204 02:53:45.460095 2023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:53:45.460204 2023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260204 02:53:45.460283 2023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260204 02:53:45.470117 2023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.189:46765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.189
--webserver_port=36571
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:53:45.471715 2023 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:53:45.473376 2023 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260204 02:53:45.481699 2040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:45.481742 2038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:45.481705 2037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:53:45.482839 2023 server_base.cc:1060] running on GCE node
I20260204 02:53:45.483635 2023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:53:45.484921 2023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:53:45.486191 2023 hybrid_clock.cc:648] HybridClock initialized: now 1770173625486118 us; error 65 us; skew 500 ppm
I20260204 02:53:45.486704 2023 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:53:45.489187 2023 webserver.cc:492] Webserver started at http://127.0.104.189:36571/ using document root <none> and password file <none>
I20260204 02:53:45.489888 2023 fs_manager.cc:362] Metadata directory not provided
I20260204 02:53:45.489996 2023 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:53:45.495205 2023 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260204 02:53:45.497983 2046 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:45.499303 2023 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260204 02:53:45.499516 2023 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
uuid: "78eb37be156448e8879b0df55832ab2d"
format_stamp: "Formatted at 2026-02-04 02:53:30 on dist-test-slave-xxhz"
I20260204 02:53:45.500276 2023 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:53:45.515372 2023 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:53:45.516327 2023 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:53:45.516606 2023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260204 02:53:45.535804 2023 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.189:46765
I20260204 02:53:45.535827 2097 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.189:46765 every 8 connection(s)
I20260204 02:53:45.537377 2023 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/info.pb
I20260204 02:53:45.543656 2098 sys_catalog.cc:263] Verifying existing consensus state
I20260204 02:53:45.546510 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 2023
I20260204 02:53:45.546954 2098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Bootstrap starting.
I20260204 02:53:45.561349 2098 log.cc:826] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Log is configured to *not* fsync() on all Append() calls
I20260204 02:53:45.588495 2098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=24 ignored=0} mutations{seen=20 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260204 02:53:45.589116 2098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Bootstrap complete.
I20260204 02:53:45.594134 2098 raft_consensus.cc:359] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:45.594813 2098 raft_consensus.cc:740] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78eb37be156448e8879b0df55832ab2d, State: Initialized, Role: FOLLOWER
I20260204 02:53:45.595448 2098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 2.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:45.596293 2103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:45.596535 2103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:45.597275 2098 sys_catalog.cc:565] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: configured and running, proceeding with master startup.
I20260204 02:53:45.606151 1205 heartbeater.cc:344] Connected to a master server at 127.0.104.189:46765
I20260204 02:53:45.630906 2118 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:45.633311 2118 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: loading cluster ID for follower catalog manager: success
I20260204 02:53:45.638823 2118 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: acquiring CA information for follower catalog manager: success
I20260204 02:53:45.634158 2023 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:53:45.641400 2023 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:53:45.644163 2118 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260204 02:53:45.648516 1066 heartbeater.cc:344] Connected to a master server at 127.0.104.189:46765
I20260204 02:53:45.651381 2063 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" instance_seqno: 1770173612117836) as {username='slave'} at 127.0.104.129:47431; Asking this server to re-register.
I20260204 02:53:45.652127 1066 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:45.653849 2063 ts_manager.cc:194] Registered new tserver with Master: 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:45.664412 1341 heartbeater.cc:344] Connected to a master server at 127.0.104.189:46765
I20260204 02:53:45.666428 2063 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" instance_seqno: 1770173613100203) as {username='slave'} at 127.0.104.131:44603; Asking this server to re-register.
I20260204 02:53:45.667088 1341 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:45.668308 2063 ts_manager.cc:194] Registered new tserver with Master: e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:45.819257 2125 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:45.819427 2125 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:45.820164 2125 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:53:45.831533 1893 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:53:45.831832 2073 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202e4a0debb24c8aa89a4724d72fcf37" candidate_term: 6 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "78eb37be156448e8879b0df55832ab2d" is_pre_election: true
I20260204 02:53:45.832536 2073 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 202e4a0debb24c8aa89a4724d72fcf37 in term 2.
I20260204 02:53:45.833129 1894 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:45.833434 2125 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260204 02:53:45.833580 2125 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:45.833693 2125 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 5 FOLLOWER]: Advancing to term 6
I20260204 02:53:45.835664 2125 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:45.836301 2125 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 election: Requested vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
I20260204 02:53:45.836799 2073 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202e4a0debb24c8aa89a4724d72fcf37" candidate_term: 6 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "78eb37be156448e8879b0df55832ab2d"
I20260204 02:53:45.837006 2073 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 2 FOLLOWER]: Advancing to term 6
W20260204 02:53:45.837459 1893 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:53:45.842566 2073 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 202e4a0debb24c8aa89a4724d72fcf37 in term 6.
I20260204 02:53:45.843012 1894 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:45.843323 2125 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 6 FOLLOWER]: Leader election won for term 6
I20260204 02:53:45.843815 2125 raft_consensus.cc:697] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 6 LEADER]: Becoming Leader. State: Replica: 202e4a0debb24c8aa89a4724d72fcf37, State: Running, Role: LEADER
I20260204 02:53:45.844293 2125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:45.846510 2129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 202e4a0debb24c8aa89a4724d72fcf37. Latest consensus state: current_term: 6 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:45.846920 2129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: LEADER
I20260204 02:53:45.847654 2131 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260204 02:53:45.849977 2131 catalog_manager.cc:679] Loaded metadata for table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]
I20260204 02:53:45.850390 2131 catalog_manager.cc:679] Loaded metadata for table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]
I20260204 02:53:45.850759 2131 catalog_manager.cc:679] Loaded metadata for table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]
I20260204 02:53:45.851092 2131 catalog_manager.cc:679] Loaded metadata for table default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796]
I20260204 02:53:45.851420 2131 catalog_manager.cc:679] Loaded metadata for table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]
I20260204 02:53:45.851799 2131 catalog_manager.cc:679] Loaded metadata for table default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2]
I20260204 02:53:45.852119 2131 catalog_manager.cc:679] Loaded metadata for table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29]
I20260204 02:53:45.852448 2131 catalog_manager.cc:679] Loaded metadata for table default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5]
I20260204 02:53:45.852787 2131 catalog_manager.cc:679] Loaded metadata for table default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f]
I20260204 02:53:45.854285 2131 tablet_loader.cc:96] loaded metadata for tablet 01792fc91bba4ba891812bd25e006268 (table default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2])
I20260204 02:53:45.854606 2131 tablet_loader.cc:96] loaded metadata for tablet 15b2b6a7198c4ebb98daf4de751911a6 (table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b])
I20260204 02:53:45.854965 2131 tablet_loader.cc:96] loaded metadata for tablet 1d7efc8694aa49e4ba8ab79721373607 (table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76])
I20260204 02:53:45.855242 2131 tablet_loader.cc:96] loaded metadata for tablet 2b883ef926504d489f3a1c0aec6db438 (table default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5])
I20260204 02:53:45.855610 2131 tablet_loader.cc:96] loaded metadata for tablet 3758b5156dd04de984cd9313459d7cfb (table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29])
I20260204 02:53:45.856376 2131 tablet_loader.cc:96] loaded metadata for tablet 8bbcb21479124152831cf7d022fa4e5f (table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793])
I20260204 02:53:45.856848 2131 tablet_loader.cc:96] loaded metadata for tablet 8f1bc638fe8e48189c15d18209efdd1d (table default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796])
I20260204 02:53:45.857314 2131 tablet_loader.cc:96] loaded metadata for tablet b321632c62d545d9b3ce83683af6de42 (table default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f])
I20260204 02:53:45.857694 2131 tablet_loader.cc:96] loaded metadata for tablet b8094eec37cd40b9b1f4931f24d824a8 (table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2])
I20260204 02:53:45.858311 2131 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260204 02:53:45.859298 2131 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:45.859393 2131 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260204 02:53:45.860831 2131 catalog_manager.cc:1514] Loading token signing keys...
I20260204 02:53:45.861651 2131 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Loaded TSK: 0
I20260204 02:53:45.862386 2131 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260204 02:53:45.863202 2131 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260204 02:53:45.864056 2131 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260204 02:53:45.921195 1902 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42974:
table { table_name: "default.Table_a16e01078d0342e7abbb4ec634032416" } modify_external_catalogs: true
I20260204 02:53:45.921537 1902 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42974:
table { table_name: "default.Table_a16e01078d0342e7abbb4ec634032416" } modify_external_catalogs: true
I20260204 02:53:45.922341 1903 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42974:
name: "default.Table_e7eb44e87e2543d69bd03454b10c8310"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:45.923539 1898 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42974:
table { table_name: "default.Table_cb3f03c12e314422b462640be08ec293" } modify_external_catalogs: true
I20260204 02:53:45.923808 1906 master_service.cc:954] ReplaceTablet: received request to replace tablet f66e2750d8c343c0bfa567fd36bc4da0 from {username='slave'} at 127.0.0.1:42974
I20260204 02:53:45.921798 1901 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42974:
name: "default.Table_6bba790402724856976fb9e10e3c48fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:45.924896 1906 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42974:
table { table_name: "default.Table_3d3a816d407e429ea458b787f879ede0" } new_table_name: "default.Table_541e82c4f45e4e74926e15734baef2cd" modify_external_catalogs: true
I20260204 02:53:45.920751 1907 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42974:
table { table_name: "default.Table_69c350f50a63410c9da5a5e7b0862cf6" } modify_external_catalogs: true
I20260204 02:53:45.922402 1905 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42974:
name: "default.Table_c26f0462e2664a018e18bf0ef613c017"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:45.923725 1898 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42974:
table { table_name: "default.Table_cb3f03c12e314422b462640be08ec293" } modify_external_catalogs: true
I20260204 02:53:45.927268 1907 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42974:
table { table_name: "default.Table_69c350f50a63410c9da5a5e7b0862cf6" } modify_external_catalogs: true
I20260204 02:53:45.922451 1900 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42974:
name: "default.Table_e971e55de1b44441b34f0913c0d665a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260204 02:53:45.930308 1959 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_3a403dbfa3044e55abc223b3acd75b6b, message={"server":"","servicePrincipal":"","db":"default","table":"table_3a403dbfa3044e55abc223b3acd75b6b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_3a403dbfa3044e55abc223b3acd75b6b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_3a403dbfa3044e55abc223b3acd75b6b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_3a403dbfa3044e55abc223b3acd75b6b\",\"kudu.table_id\":\"333921075666492fb4e06fa35e04487b\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_0312705ce5e140d4bcf88a92cab2bb7b, message={"server":"","servicePrincipal":"","db":"default","table":"table_0312705ce5e140d4bcf88a92cab2bb7b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0312705ce5e140d4bcf88a92cab2bb7b\",\"kudu.table_id\":\"8df7ca3e2276436e8bdb4480fc3dd711\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20260204 02:53:45.922825 1899 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42974:
name: "default.Table_3f77bbc9c07244dcb50a3b82a1fc8f1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:45.922923 1904 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42974:
name: "default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fb678dfedcd google::LogMessage::Fail() at ??:0
@ 0x7fb678e02b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb678dfe7cc google::LogMessage::Flush() at ??:0
@ 0x7fb678dfea11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fb68f8327f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fb68f8310a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fb68f838d1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fb68f838b99 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7fb67bb73f99 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb67f1806db start_thread at ??:0
@ 0x7fb677caf71f clone at ??:0
I20260204 02:53:46.112528 2063 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" instance_seqno: 1770173612604702) as {username='slave'} at 127.0.104.130:49249; Asking this server to re-register.
I20260204 02:53:46.113474 1205 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:46.115156 2063 ts_manager.cc:194] Registered new tserver with Master: 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:53:46.127211 1091 connection.cc:537] client connection to 127.0.104.188:38789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:53:46.128304 1346 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.188:38789)
W20260204 02:53:46.128830 1346 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.188:38789)
W20260204 02:53:46.128991 1346 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.188:38789)
W20260204 02:53:46.129341 1346 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.188:38789)
I20260204 02:53:46.453827 2149 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:46.454058 2149 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:46.455741 2149 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:46.456619 2049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:46.457758 2047 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
W20260204 02:53:46.457945 2049 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:53:46.458086 2049 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:46.458376 2149 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260204 02:53:46.971338 2158 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:46.971562 2158 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:46.972184 2158 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:46.973953 2047 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
W20260204 02:53:46.974256 2049 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:53:46.974392 2049 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:46.974637 2158 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260204 02:53:47.595880 2165 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:47.596029 2165 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:47.596534 2165 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:47.598309 2047 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
W20260204 02:53:47.598528 2049 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:53:47.598634 2049 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:47.598858 2165 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260204 02:53:47.808164 418 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task6hEvLx/build/asan/bin/kudu pid 2023
I20260204 02:53:47.815856 2093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260204 02:53:47.925956 418 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6hEvLx/build/asan/bin/kudu with pid 2023
W20260204 02:53:47.945317 955 connection.cc:537] client connection to 127.0.104.189:46765 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260204 02:53:48.695585 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.189:46765
--webserver_interface=127.0.104.189
--webserver_port=36571
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260204 02:53:49.021459 2188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:53:49.021749 2188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:53:49.021831 2188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260204 02:53:49.029713 2188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260204 02:53:49.029816 2188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:53:49.029909 2188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260204 02:53:49.029984 2188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260204 02:53:49.039259 2188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.189:46765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.189
--webserver_port=36571
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:53:49.040876 2188 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:53:49.042586 2188 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260204 02:53:49.050693 2196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:49.050771 2197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:49.051224 2199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:53:49.052183 2188 server_base.cc:1060] running on GCE node
I20260204 02:53:49.053036 2188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:53:49.054490 2188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:53:49.055714 2188 hybrid_clock.cc:648] HybridClock initialized: now 1770173629055681 us; error 49 us; skew 500 ppm
I20260204 02:53:49.056047 2188 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:53:49.058106 2188 webserver.cc:492] Webserver started at http://127.0.104.189:36571/ using document root <none> and password file <none>
I20260204 02:53:49.058863 2188 fs_manager.cc:362] Metadata directory not provided
I20260204 02:53:49.058993 2188 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:53:49.064095 2188 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260204 02:53:49.067075 2205 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:49.068671 2188 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260204 02:53:49.068820 2188 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
uuid: "78eb37be156448e8879b0df55832ab2d"
format_stamp: "Formatted at 2026-02-04 02:53:30 on dist-test-slave-xxhz"
I20260204 02:53:49.069433 2188 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:53:49.111420 2188 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:53:49.112287 2188 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:53:49.112564 2188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260204 02:53:49.144353 2188 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.189:46765
I20260204 02:53:49.144384 2258 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.189:46765 every 8 connection(s)
I20260204 02:53:49.145906 2188 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/info.pb
I20260204 02:53:49.153911 2259 sys_catalog.cc:263] Verifying existing consensus state
I20260204 02:53:49.155501 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 2188
I20260204 02:53:49.159857 2259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Bootstrap starting.
I20260204 02:53:49.182909 1205 heartbeater.cc:344] Connected to a master server at 127.0.104.189:46765
I20260204 02:53:49.184509 2259 log.cc:826] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Log is configured to *not* fsync() on all Append() calls
I20260204 02:53:49.187178 1341 heartbeater.cc:344] Connected to a master server at 127.0.104.189:46765
I20260204 02:53:49.212067 2259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=24 ignored=0} mutations{seen=20 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260204 02:53:49.212695 2259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Bootstrap complete.
I20260204 02:53:49.217694 2259 raft_consensus.cc:359] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:49.218259 2259 raft_consensus.cc:740] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78eb37be156448e8879b0df55832ab2d, State: Initialized, Role: FOLLOWER
I20260204 02:53:49.218832 2259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 2.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:49.219661 2267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:49.219949 2267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:49.220618 2259 sys_catalog.cc:565] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: configured and running, proceeding with master startup.
I20260204 02:53:49.233842 2281 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:49.234107 2281 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: loading cluster ID for follower catalog manager: success
I20260204 02:53:49.242251 2281 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: acquiring CA information for follower catalog manager: success
I20260204 02:53:49.245500 2281 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260204 02:53:49.244787 2188 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:53:49.249254 2188 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:53:49.256860 2223 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" instance_seqno: 1770173612604702) as {username='slave'} at 127.0.104.130:58637; Asking this server to re-register.
I20260204 02:53:49.257555 1205 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:49.259296 2223 ts_manager.cc:194] Registered new tserver with Master: 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:49.436647 1066 heartbeater.cc:344] Connected to a master server at 127.0.104.189:46765
I20260204 02:53:49.438479 2223 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" instance_seqno: 1770173612117836) as {username='slave'} at 127.0.104.129:41565; Asking this server to re-register.
I20260204 02:53:49.439043 1066 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:49.440312 2223 ts_manager.cc:194] Registered new tserver with Master: 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:49.690171 2223 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" instance_seqno: 1770173613100203) as {username='slave'} at 127.0.104.131:59579; Asking this server to re-register.
I20260204 02:53:49.690773 1341 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:49.692080 2223 ts_manager.cc:194] Registered new tserver with Master: e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:49.724176 2288 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:49.724469 2288 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:49.726208 2288 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:49.727077 2208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:49.728224 2206 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
W20260204 02:53:49.728447 2208 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:53:49.728607 2208 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:49.728971 2288 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260204 02:53:49.732111 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 6: this message will repeat every 5th retry.
W20260204 02:53:50.366312 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 6: this message will repeat every 5th retry.
I20260204 02:53:50.374828 2304 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:50.374976 2304 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:50.375727 2304 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:50.377437 2208 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:50.377586 2206 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:50.377712 2206 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:50.377921 2304 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260204 02:53:50.751919 954 consensus_peers.cc:597] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027. This is attempt 6: this message will repeat every 5th retry.
I20260204 02:53:51.426108 2317 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:51.426359 2317 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:51.426997 2317 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:51.429165 2208 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:51.429368 2206 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:51.429534 2206 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:51.429772 2317 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260204 02:53:52.434942 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.188:38789
--webserver_interface=127.0.104.188
--webserver_port=40055
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260204 02:53:52.602331 2335 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:52.602564 2335 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:52.603220 2335 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:52.605360 2208 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:52.605515 2206 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:52.605652 2206 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:52.605897 2335 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260204 02:53:52.748217 2331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:53:52.748883 2331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:53:52.749011 2331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260204 02:53:52.761583 2331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260204 02:53:52.761698 2331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:53:52.761787 2331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260204 02:53:52.761852 2331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260204 02:53:52.776697 2331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.188:38789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.188
--webserver_port=40055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:53:52.778972 2331 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:53:52.781492 2331 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260204 02:53:52.791529 2343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:52.791942 2346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:52.792527 2344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:53:52.793190 2331 server_base.cc:1060] running on GCE node
I20260204 02:53:52.794170 2331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:53:52.796164 2331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:53:52.797456 2331 hybrid_clock.cc:648] HybridClock initialized: now 1770173632797381 us; error 56 us; skew 500 ppm
I20260204 02:53:52.797894 2331 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:53:52.799882 2331 webserver.cc:492] Webserver started at http://127.0.104.188:40055/ using document root <none> and password file <none>
I20260204 02:53:52.800537 2331 fs_manager.cc:362] Metadata directory not provided
I20260204 02:53:52.800621 2331 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:53:52.805579 2331 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260204 02:53:52.808240 2352 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:52.809501 2331 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260204 02:53:52.809660 2331 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
uuid: "202e4a0debb24c8aa89a4724d72fcf37"
format_stamp: "Formatted at 2026-02-04 02:53:31 on dist-test-slave-xxhz"
I20260204 02:53:52.810264 2331 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:53:52.822319 2331 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:53:52.823149 2331 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:53:52.823565 2331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260204 02:53:52.842453 2331 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.188:38789
I20260204 02:53:52.842519 2404 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.188:38789 every 8 connection(s)
I20260204 02:53:52.844864 2331 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
I20260204 02:53:52.852149 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 2331
I20260204 02:53:52.853742 2407 sys_catalog.cc:263] Verifying existing consensus state
I20260204 02:53:52.860554 2407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap starting.
I20260204 02:53:52.863123 1064 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:52.874760 2407 log.cc:826] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Log is configured to *not* fsync() on all Append() calls
I20260204 02:53:52.905916 2407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap replayed 1/1 log segments. Stats: ops{read=43 overwritten=0 applied=42 ignored=0} inserts{seen=28 ignored=0} mutations{seen=32 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260204 02:53:52.906595 2407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap complete.
I20260204 02:53:52.911345 2407 raft_consensus.cc:359] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:52.912236 2407 raft_consensus.cc:740] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202e4a0debb24c8aa89a4724d72fcf37, State: Initialized, Role: FOLLOWER
I20260204 02:53:52.912822 2407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 6.43, Last appended by leader: 43, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:52.913530 2412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:52.913748 2412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:52.914525 2407 sys_catalog.cc:565] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: configured and running, proceeding with master startup.
I20260204 02:53:52.927790 2426 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:52.927898 2426 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: loading cluster ID for follower catalog manager: success
I20260204 02:53:52.933199 2426 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: acquiring CA information for follower catalog manager: success
I20260204 02:53:52.936966 2426 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260204 02:53:52.938241 2331 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:53:52.943113 2331 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:53:53.002118 1203 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:53.004375 2368 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" instance_seqno: 1770173612604702) as {username='slave'} at 127.0.104.130:58193; Asking this server to re-register.
I20260204 02:53:53.005200 1203 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:53.006842 2368 ts_manager.cc:194] Registered new tserver with Master: 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:53.148988 2368 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" instance_seqno: 1770173612117836) as {username='slave'} at 127.0.104.129:59685; Asking this server to re-register.
I20260204 02:53:53.149675 1064 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:53.151052 2368 ts_manager.cc:194] Registered new tserver with Master: 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:53.179303 1339 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:53.180903 2368 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" instance_seqno: 1770173613100203) as {username='slave'} at 127.0.104.131:49391; Asking this server to re-register.
I20260204 02:53:53.181536 1339 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:53.183081 2368 ts_manager.cc:194] Registered new tserver with Master: e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:53.430977 2437 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:53.431187 2437 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:53.432880 2437 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:53:53.433578 2355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:53.434942 2355 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:53:53.438779 2234 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202e4a0debb24c8aa89a4724d72fcf37" candidate_term: 7 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "78eb37be156448e8879b0df55832ab2d" is_pre_election: true
I20260204 02:53:53.439134 2234 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 202e4a0debb24c8aa89a4724d72fcf37 in term 6.
I20260204 02:53:53.439759 2356 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:53.440100 2437 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260204 02:53:53.440259 2437 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:53.440366 2437 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 6 FOLLOWER]: Advancing to term 7
I20260204 02:53:53.443588 2437 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:53.444154 2437 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 7 election: Requested vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
I20260204 02:53:53.444610 2234 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202e4a0debb24c8aa89a4724d72fcf37" candidate_term: 7 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "78eb37be156448e8879b0df55832ab2d"
I20260204 02:53:53.444802 2234 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 6 FOLLOWER]: Advancing to term 7
W20260204 02:53:53.445145 2355 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:53:53.448292 2234 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 202e4a0debb24c8aa89a4724d72fcf37 in term 7.
I20260204 02:53:53.448669 2356 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:53.448906 2437 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 7 FOLLOWER]: Leader election won for term 7
I20260204 02:53:53.449254 2437 raft_consensus.cc:697] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 7 LEADER]: Becoming Leader. State: Replica: 202e4a0debb24c8aa89a4724d72fcf37, State: Running, Role: LEADER
I20260204 02:53:53.449645 2437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 6.43, Last appended by leader: 43, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:53.452147 2441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 202e4a0debb24c8aa89a4724d72fcf37. Latest consensus state: current_term: 7 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:53.452445 2441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: LEADER
I20260204 02:53:53.453051 2443 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260204 02:53:53.455372 2443 catalog_manager.cc:679] Loaded metadata for table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]
I20260204 02:53:53.455830 2443 catalog_manager.cc:679] Loaded metadata for table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]
I20260204 02:53:53.456142 2443 catalog_manager.cc:679] Loaded metadata for table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]
I20260204 02:53:53.456435 2443 catalog_manager.cc:679] Loaded metadata for table default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796]
I20260204 02:53:53.456698 2443 catalog_manager.cc:679] Loaded metadata for table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]
I20260204 02:53:53.456969 2443 catalog_manager.cc:679] Loaded metadata for table default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2]
I20260204 02:53:53.457392 2443 catalog_manager.cc:679] Loaded metadata for table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29]
I20260204 02:53:53.457804 2443 catalog_manager.cc:679] Loaded metadata for table default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5]
I20260204 02:53:53.458202 2443 catalog_manager.cc:679] Loaded metadata for table default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f]
I20260204 02:53:53.460263 2443 tablet_loader.cc:96] loaded metadata for tablet 01792fc91bba4ba891812bd25e006268 (table default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2])
I20260204 02:53:53.460675 2443 tablet_loader.cc:96] loaded metadata for tablet 15b2b6a7198c4ebb98daf4de751911a6 (table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b])
I20260204 02:53:53.461081 2443 tablet_loader.cc:96] loaded metadata for tablet 1d7efc8694aa49e4ba8ab79721373607 (table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76])
I20260204 02:53:53.461418 2443 tablet_loader.cc:96] loaded metadata for tablet 2b883ef926504d489f3a1c0aec6db438 (table default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5])
I20260204 02:53:53.461745 2443 tablet_loader.cc:96] loaded metadata for tablet 3758b5156dd04de984cd9313459d7cfb (table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29])
I20260204 02:53:53.462682 2443 tablet_loader.cc:96] loaded metadata for tablet 8bbcb21479124152831cf7d022fa4e5f (table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793])
I20260204 02:53:53.463220 2443 tablet_loader.cc:96] loaded metadata for tablet 8f1bc638fe8e48189c15d18209efdd1d (table default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796])
I20260204 02:53:53.463734 2443 tablet_loader.cc:96] loaded metadata for tablet b321632c62d545d9b3ce83683af6de42 (table default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f])
I20260204 02:53:53.464046 2443 tablet_loader.cc:96] loaded metadata for tablet b8094eec37cd40b9b1f4931f24d824a8 (table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2])
I20260204 02:53:53.464790 2443 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260204 02:53:53.465790 2443 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:53.465858 2443 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260204 02:53:53.467080 2443 catalog_manager.cc:1514] Loading token signing keys...
I20260204 02:53:53.468091 2443 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Loaded TSK: 0
I20260204 02:53:53.468825 2443 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260204 02:53:53.469607 2443 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260204 02:53:53.470346 2443 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
W20260204 02:53:53.515305 1093 consensus_peers.cc:597] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969. This is attempt 6: this message will repeat every 5th retry.
I20260204 02:53:53.522918 1136 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c77d21f66a640d88bf1865d6176c969 with delete_type TABLET_DATA_DELETED (replaced by tablet 3758b5156dd04de984cd9313459d7cfb) from {username='slave'} at 127.0.0.1:43510
I20260204 02:53:53.523586 2445 tablet_replica.cc:333] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8: stopping tablet replica
I20260204 02:53:53.523844 2445 raft_consensus.cc:2243] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 2 LEADER]: Raft consensus shutting down.
I20260204 02:53:53.524251 2445 raft_consensus.cc:2272] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:53.524749 2445 ts_tablet_manager.cc:1916] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:53.528883 2445 ts_tablet_manager.cc:1929] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260204 02:53:53.529009 2445 log.cc:1199] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/wal/wals/8c77d21f66a640d88bf1865d6176c969
I20260204 02:53:53.529407 2445 ts_tablet_manager.cc:1950] T 8c77d21f66a640d88bf1865d6176c969 P 4dc2992e324049698330803c9ed2e3f8: Deleting consensus metadata
I20260204 02:53:53.530333 2353 catalog_manager.cc:4985] TS 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): tablet 8c77d21f66a640d88bf1865d6176c969 (table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29]) successfully deleted
I20260204 02:53:53.588687 2369 master_service.cc:954] ReplaceTablet: received request to replace tablet 9a80cbfc27b54a5d945683c39020a027 from {username='slave'} at 127.0.0.1:39028
I20260204 02:53:53.588872 2367 master_service.cc:954] ReplaceTablet: received request to replace tablet 1d7efc8694aa49e4ba8ab79721373607 from {username='slave'} at 127.0.0.1:39028
I20260204 02:53:53.588958 2366 master_service.cc:954] ReplaceTablet: received request to replace tablet b321632c62d545d9b3ce83683af6de42 from {username='slave'} at 127.0.0.1:39028
I20260204 02:53:53.588735 2368 master_service.cc:954] ReplaceTablet: received request to replace tablet 8c77d21f66a640d88bf1865d6176c969 from {username='slave'} at 127.0.0.1:39028
I20260204 02:53:53.589748 2364 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39028:
table { table_name: "default.Table_f734190e49874f9fb2c8dee62aa9e70b" } new_table_name: "default.Table_f9d078d036094cf486755f573c75e8b6" modify_external_catalogs: true
I20260204 02:53:53.590165 2363 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39028:
table { table_name: "default.Table_db7b7860261b41c692b36ca95b2d89e2" } new_table_name: "default.Table_667ea74f761c447c8538d777329b0b9f" modify_external_catalogs: true
I20260204 02:53:53.590543 2368 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39028:
name: "default.Table_c26f0462e2664a018e18bf0ef613c017"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:53.590605 2361 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39028:
name: "default.Table_e7eb44e87e2543d69bd03454b10c8310"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:53.590775 2365 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39028:
name: "default.Table_bdc0d01a86b043beb3441b2078794538"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:53.591398 2360 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39028:
name: "default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:53.591917 2369 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39028:
table { table_name: "default.Table_a16e01078d0342e7abbb4ec634032416" } modify_external_catalogs: true
I20260204 02:53:53.592163 2369 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39028:
table { table_name: "default.Table_a16e01078d0342e7abbb4ec634032416" } modify_external_catalogs: true
I20260204 02:53:53.589775 2362 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39028:
table { table_name: "default.Table_842bb3fe3b2944498784e63317dd1907" } new_table_name: "default.Table_fb4d5c02e3f443afb1f2f219731403b1" modify_external_catalogs: true
F20260204 02:53:53.597455 2425 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_3a403dbfa3044e55abc223b3acd75b6b, message={"server":"","servicePrincipal":"","db":"default","table":"table_3a403dbfa3044e55abc223b3acd75b6b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_3a403dbfa3044e55abc223b3acd75b6b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_3a403dbfa3044e55abc223b3acd75b6b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_3a403dbfa3044e55abc223b3acd75b6b\",\"kudu.table_id\":\"333921075666492fb4e06fa35e04487b\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_0312705ce5e140d4bcf88a92cab2bb7b, message={"server":"","servicePrincipal":"","db":"default","table":"table_0312705ce5e140d4bcf88a92cab2bb7b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0312705ce5e140d4bcf88a92cab2bb7b\",\"kudu.table_id\":\"8df7ca3e2276436e8bdb4480fc3dd711\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
W20260204 02:53:53.600033 2355 consensus_peers.cc:597] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 -> Peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Couldn't send request to peer 3e85e1dc6a5d4568ad365ba1d77398ff. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f9fbb305dcd google::LogMessage::Fail() at ??:0
I20260204 02:53:53.600724 2234 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 7 FOLLOWER]: Refusing update from remote peer 202e4a0debb24c8aa89a4724d72fcf37: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 7 index: 45. (index mismatch)
@ 0x7f9fbb309b93 google::LogMessage::SendToLog() at ??:0
I20260204 02:53:53.601511 2437 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 34, Time since last communication: 0.000s
@ 0x7f9fbb3057cc google::LogMessage::Flush() at ??:0
@ 0x7f9fbb305a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9fd1d397f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f9fd1d380a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f9fd1d3fd1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260204 02:53:53.620664 2451 mvcc.cc:204] Tried to move back new op lower bound from 7250631146595352576 to 7250631127872675840. Current Snapshot: MvccSnapshot[applied={T|T < 7250631146595352576 or (T in {7250631146595352576})}]
@ 0x7f9fd1d3fb99 std::_Function_handler<>::_M_invoke() at ??:0
I20260204 02:53:53.623016 2454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:53.623342 2454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:53.623709 2449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 202e4a0debb24c8aa89a4724d72fcf37. Latest consensus state: current_term: 7 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:53.623965 2449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: FOLLOWER
@ 0xa8970d std::function<>::operator()() at ??:?
I20260204 02:53:53.624221 2458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:53.624449 2458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:53.625749 2447 mvcc.cc:204] Tried to move back new op lower bound from 7250631203214143488 to 7250631171461468160. Current Snapshot: MvccSnapshot[applied={T|T < 7250631203214143488}]
I20260204 02:53:53.627592 2449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:53.627737 2449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:53.627976 2458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:53.628100 2458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f9fbe07af99 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9fc16876db start_thread at ??:0
@ 0x7f9fba1b671f clone at ??:0
W20260204 02:53:53.798233 1227 connection.cc:537] client connection to 127.0.104.188:38789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:53:53.799839 1346 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.188:38789)
W20260204 02:53:53.800148 1346 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.188:38789)
W20260204 02:53:53.800328 1346 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.188:38789)
W20260204 02:53:53.800592 1346 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.188:38789)
W20260204 02:53:53.800997 952 connection.cc:537] client connection to 127.0.104.188:38789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260204 02:53:54.019129 1203 heartbeater.cc:499] Master 127.0.104.188:38789 was elected leader, sending a full tablet report...
I20260204 02:53:54.126029 2458 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 202e4a0debb24c8aa89a4724d72fcf37)
I20260204 02:53:54.126354 2458 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:54.127040 2458 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:54.129407 2208 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:54.129545 2206 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:54.129703 2206 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:54.129917 2458 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260204 02:53:54.250486 2188 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:53:54.250676 2188 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:53:54.877133 2490 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 202e4a0debb24c8aa89a4724d72fcf37)
I20260204 02:53:54.877339 2490 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:54.877914 2490 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:54.878602 2208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260204 02:53:54.879545 2206 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
W20260204 02:53:54.879712 2208 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:53:54.879801 2208 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:54.879980 2490 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260204 02:53:55.580235 2499 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 202e4a0debb24c8aa89a4724d72fcf37)
I20260204 02:53:55.580440 2499 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:55.581039 2499 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:55.582809 2208 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:55.582973 2206 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:55.583078 2206 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:55.583289 2499 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260204 02:53:56.003398 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.188:38789
--webserver_interface=127.0.104.188
--webserver_port=40055
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260204 02:53:56.339202 2505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:53:56.339569 2505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:53:56.339653 2505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260204 02:53:56.347260 2505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260204 02:53:56.347337 2505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:53:56.347400 2505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260204 02:53:56.347456 2505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260204 02:53:56.356739 2505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.188:38789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.188
--webserver_port=40055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:53:56.358295 2505 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:53:56.360200 2505 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260204 02:53:56.364400 2511 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:53:56.364586 2511 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:56.365665 2511 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:53:56.367295 2516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:56.367630 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:53:56.367794 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:53:56.367926 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:53:56.368225 2511 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260204 02:53:56.368220 2517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:53:56.368575 2519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:53:56.369517 2505 server_base.cc:1060] running on GCE node
I20260204 02:53:56.370102 2505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:53:56.371340 2505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:53:56.372536 2505 hybrid_clock.cc:648] HybridClock initialized: now 1770173636372485 us; error 48 us; skew 500 ppm
I20260204 02:53:56.372881 2505 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:53:56.374954 2505 webserver.cc:492] Webserver started at http://127.0.104.188:40055/ using document root <none> and password file <none>
I20260204 02:53:56.375689 2505 fs_manager.cc:362] Metadata directory not provided
I20260204 02:53:56.375800 2505 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:53:56.380964 2505 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260204 02:53:56.383667 2525 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:53:56.384833 2505 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260204 02:53:56.384991 2505 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
uuid: "202e4a0debb24c8aa89a4724d72fcf37"
format_stamp: "Formatted at 2026-02-04 02:53:31 on dist-test-slave-xxhz"
I20260204 02:53:56.385649 2505 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:53:56.397696 2505 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:53:56.398440 2505 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:53:56.398689 2505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260204 02:53:56.418092 2505 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.188:38789
I20260204 02:53:56.418133 2576 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.188:38789 every 8 connection(s)
I20260204 02:53:56.419615 2505 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
I20260204 02:53:56.421017 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 2505
I20260204 02:53:56.428714 2577 sys_catalog.cc:263] Verifying existing consensus state
I20260204 02:53:56.432031 2577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap starting.
I20260204 02:53:56.443198 2577 log.cc:826] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Log is configured to *not* fsync() on all Append() calls
I20260204 02:53:56.475140 2577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap replayed 1/1 log segments. Stats: ops{read=46 overwritten=0 applied=44 ignored=0} inserts{seen=28 ignored=0} mutations{seen=32 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20260204 02:53:56.475817 2577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap complete.
I20260204 02:53:56.481859 2577 raft_consensus.cc:359] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 7 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:56.485646 2577 raft_consensus.cc:740] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202e4a0debb24c8aa89a4724d72fcf37, State: Initialized, Role: FOLLOWER
I20260204 02:53:56.486399 2577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 7.46, Last appended by leader: 46, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:56.487145 2582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:56.487397 2582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:56.488394 2577 sys_catalog.cc:565] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: configured and running, proceeding with master startup.
I20260204 02:53:56.500692 2596 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:56.500896 2596 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: loading cluster ID for follower catalog manager: success
I20260204 02:53:56.506311 2596 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: acquiring CA information for follower catalog manager: success
I20260204 02:53:56.510236 2596 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260204 02:53:56.511547 2505 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:53:56.515362 2505 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:53:56.661736 1203 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:56.661985 1203 heartbeater.cc:499] Master 127.0.104.188:38789 was elected leader, sending a full tablet report...
I20260204 02:53:56.666301 2542 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" instance_seqno: 1770173612604702) as {username='slave'} at 127.0.104.130:52601; Asking this server to re-register.
I20260204 02:53:56.667201 1203 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:56.669255 2542 ts_manager.cc:194] Registered new tserver with Master: 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:53:56.812033 1339 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:56.813328 2542 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" instance_seqno: 1770173613100203) as {username='slave'} at 127.0.104.131:48369; Asking this server to re-register.
I20260204 02:53:56.813906 1339 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:56.815512 2542 ts_manager.cc:194] Registered new tserver with Master: e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:53:56.817418 1064 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:53:56.818783 2542 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" instance_seqno: 1770173612117836) as {username='slave'} at 127.0.104.129:40167; Asking this server to re-register.
I20260204 02:53:56.819322 1064 heartbeater.cc:461] Registering TS with master...
I20260204 02:53:56.820544 2542 ts_manager.cc:194] Registered new tserver with Master: 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:56.950279 2582 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:56.950518 2582 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:56.952240 2582 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:53:56.953053 2528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:56.954607 2528 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:53:56.958523 2234 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202e4a0debb24c8aa89a4724d72fcf37" candidate_term: 8 candidate_status { last_received { term: 7 index: 46 } } ignore_live_leader: false dest_uuid: "78eb37be156448e8879b0df55832ab2d" is_pre_election: true
I20260204 02:53:56.958748 2234 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 202e4a0debb24c8aa89a4724d72fcf37 in term 7.
I20260204 02:53:56.959345 2529 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:56.959765 2582 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260204 02:53:56.959908 2582 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:56.960021 2582 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 7 FOLLOWER]: Advancing to term 8
I20260204 02:53:56.963197 2582 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:56.963797 2582 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 8 election: Requested vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
I20260204 02:53:56.964346 2234 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202e4a0debb24c8aa89a4724d72fcf37" candidate_term: 8 candidate_status { last_received { term: 7 index: 46 } } ignore_live_leader: false dest_uuid: "78eb37be156448e8879b0df55832ab2d"
I20260204 02:53:56.964541 2234 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 7 FOLLOWER]: Advancing to term 8
W20260204 02:53:56.964941 2528 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:53:56.966394 2234 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 202e4a0debb24c8aa89a4724d72fcf37 in term 8.
I20260204 02:53:56.966771 2529 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:56.967068 2582 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 8 FOLLOWER]: Leader election won for term 8
I20260204 02:53:56.967505 2582 raft_consensus.cc:697] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 8 LEADER]: Becoming Leader. State: Replica: 202e4a0debb24c8aa89a4724d72fcf37, State: Running, Role: LEADER
I20260204 02:53:56.968031 2582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 7.46, Last appended by leader: 46, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:56.970438 2610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 202e4a0debb24c8aa89a4724d72fcf37. Latest consensus state: current_term: 8 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:56.970687 2610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: LEADER
W20260204 02:53:57.441706 2528 consensus_peers.cc:597] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 -> Peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Couldn't send request to peer 3e85e1dc6a5d4568ad365ba1d77398ff. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260204 02:53:57.533877 2234 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Refusing update from remote peer 202e4a0debb24c8aa89a4724d72fcf37: Log matching property violated. Preceding OpId in replica: term: 7 index: 46. Preceding OpId from leader: term: 8 index: 47. (index mismatch)
I20260204 02:53:57.534561 2610 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20260204 02:53:57.536445 2617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 202e4a0debb24c8aa89a4724d72fcf37. Latest consensus state: current_term: 8 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:57.536680 2617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:57.539130 2617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:57.539333 2617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:53:57.540205 2624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "202e4a0debb24c8aa89a4724d72fcf37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:53:57.540385 2624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: LEADER
I20260204 02:53:57.616420 2612 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260204 02:53:57.618798 2612 catalog_manager.cc:679] Loaded metadata for table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]
I20260204 02:53:57.619175 2612 catalog_manager.cc:679] Loaded metadata for table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]
I20260204 02:53:57.619505 2612 catalog_manager.cc:679] Loaded metadata for table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]
I20260204 02:53:57.619885 2612 catalog_manager.cc:679] Loaded metadata for table default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796]
I20260204 02:53:57.620169 2612 catalog_manager.cc:679] Loaded metadata for table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]
I20260204 02:53:57.620527 2612 catalog_manager.cc:679] Loaded metadata for table default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2]
I20260204 02:53:57.620834 2612 catalog_manager.cc:679] Loaded metadata for table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29]
I20260204 02:53:57.621183 2612 catalog_manager.cc:679] Loaded metadata for table default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5]
I20260204 02:53:57.621649 2612 catalog_manager.cc:679] Loaded metadata for table default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f]
I20260204 02:53:57.624037 2612 tablet_loader.cc:96] loaded metadata for tablet 01792fc91bba4ba891812bd25e006268 (table default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2])
I20260204 02:53:57.624367 2612 tablet_loader.cc:96] loaded metadata for tablet 15b2b6a7198c4ebb98daf4de751911a6 (table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b])
I20260204 02:53:57.624841 2612 tablet_loader.cc:96] loaded metadata for tablet 2b883ef926504d489f3a1c0aec6db438 (table default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5])
I20260204 02:53:57.625113 2612 tablet_loader.cc:96] loaded metadata for tablet 3758b5156dd04de984cd9313459d7cfb (table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29])
I20260204 02:53:57.625351 2612 tablet_loader.cc:96] loaded metadata for tablet 38b39855b7dd4ea58a1cfc97849b088d (table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76])
I20260204 02:53:57.626152 2612 tablet_loader.cc:96] loaded metadata for tablet 8bbcb21479124152831cf7d022fa4e5f (table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793])
I20260204 02:53:57.626633 2612 tablet_loader.cc:96] loaded metadata for tablet 8f1bc638fe8e48189c15d18209efdd1d (table default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796])
I20260204 02:53:57.627068 2612 tablet_loader.cc:96] loaded metadata for tablet a1f6cc066dad428ead30300716e58dfd (table default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f])
I20260204 02:53:57.627547 2612 tablet_loader.cc:96] loaded metadata for tablet b8094eec37cd40b9b1f4931f24d824a8 (table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2])
I20260204 02:53:57.628103 2612 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260204 02:53:57.629043 2612 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:53:57.629169 2612 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260204 02:53:57.630403 2612 catalog_manager.cc:1514] Loading token signing keys...
I20260204 02:53:57.631259 2612 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Loaded TSK: 0
I20260204 02:53:57.632133 2612 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260204 02:53:57.633003 2612 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260204 02:53:57.633828 2612 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260204 02:53:57.690222 997 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a80cbfc27b54a5d945683c39020a027 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d7efc8694aa49e4ba8ab79721373607) from {username='slave'} at 127.0.0.1:56548
I20260204 02:53:57.690888 2628 tablet_replica.cc:333] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: stopping tablet replica
I20260204 02:53:57.691082 2628 raft_consensus.cc:2243] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 LEADER]: Raft consensus shutting down.
I20260204 02:53:57.691644 2628 raft_consensus.cc:2272] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:57.692174 2628 ts_tablet_manager.cc:1916] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:57.696364 2628 ts_tablet_manager.cc:1929] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260204 02:53:57.696485 2628 log.cc:1199] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/wal/wals/9a80cbfc27b54a5d945683c39020a027
I20260204 02:53:57.696856 2628 ts_tablet_manager.cc:1950] T 9a80cbfc27b54a5d945683c39020a027 P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting consensus metadata
I20260204 02:53:57.697811 2526 catalog_manager.cc:4985] TS 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): tablet 9a80cbfc27b54a5d945683c39020a027 (table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]) successfully deleted
I20260204 02:53:57.704743 2542 master_service.cc:954] ReplaceTablet: received request to replace tablet 8bbcb21479124152831cf7d022fa4e5f from {username='slave'} at 127.0.0.1:33206
I20260204 02:53:57.705771 2541 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33206:
table { table_name: "default.Table_3a403dbfa3044e55abc223b3acd75b6b" } new_table_name: "default.Table_7021e56527ac45df985d01556abaeeb8" modify_external_catalogs: true
I20260204 02:53:57.706054 2538 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33206:
table { table_name: "default.Table_79ae542654da4da5aba4d8c38d6abae7" } modify_external_catalogs: true
I20260204 02:53:57.706276 2538 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33206:
table { table_name: "default.Table_79ae542654da4da5aba4d8c38d6abae7" } modify_external_catalogs: true
I20260204 02:53:57.706130 2539 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33206:
name: "default.Table_a2c8b05d2d724f50aaf92bddb8a266e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:57.706508 2536 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33206:
table { table_name: "default.Table_3d3a816d407e429ea458b787f879ede0" } new_table_name: "default.Table_541e82c4f45e4e74926e15734baef2cd" modify_external_catalogs: true
I20260204 02:53:57.706848 2535 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33206:
table { table_name: "default.Table_cb3f03c12e314422b462640be08ec293" } modify_external_catalogs: true
I20260204 02:53:57.706801 2540 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33206:
name: "default.Table_b896155e926a4679a475f25f74e33530"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:57.707069 2535 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33206:
table { table_name: "default.Table_cb3f03c12e314422b462640be08ec293" } modify_external_catalogs: true
I20260204 02:53:57.707103 2537 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33206:
name: "default.Table_237c40ddb38c410c9ce533b72060712f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:57.708014 2533 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33206:
name: "default.Table_bdc0d01a86b043beb3441b2078794538"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:53:57.708387 2534 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33206:
table { table_name: "default.Table_f734190e49874f9fb2c8dee62aa9e70b" } new_table_name: "default.Table_f9d078d036094cf486755f573c75e8b6" modify_external_catalogs: true
F20260204 02:53:57.713445 2595 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_3a403dbfa3044e55abc223b3acd75b6b, message={"server":"","servicePrincipal":"","db":"default","table":"table_3a403dbfa3044e55abc223b3acd75b6b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_3a403dbfa3044e55abc223b3acd75b6b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_3a403dbfa3044e55abc223b3acd75b6b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_3a403dbfa3044e55abc223b3acd75b6b\",\"kudu.table_id\":\"333921075666492fb4e06fa35e04487b\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_0312705ce5e140d4bcf88a92cab2bb7b, message={"server":"","servicePrincipal":"","db":"default","table":"table_0312705ce5e140d4bcf88a92cab2bb7b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0312705ce5e140d4bcf88a92cab2bb7b\",\"kudu.table_id\":\"8df7ca3e2276436e8bdb4480fc3dd711\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20260204 02:53:57.715281 2542 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Sending DeleteTablet for 3 replicas of tablet 8bbcb21479124152831cf7d022fa4e5f
@ 0x7fde467eedcd google::LogMessage::Fail() at ??:0
I20260204 02:53:57.716970 997 tablet_service.cc:1552] Processing DeleteTablet for tablet 8bbcb21479124152831cf7d022fa4e5f with delete_type TABLET_DATA_DELETED (replaced by tablet ad418256b01c415aa306182f3eccd69b) from {username='slave'} at 127.0.0.1:56548
I20260204 02:53:57.717250 2628 tablet_replica.cc:333] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: stopping tablet replica
I20260204 02:53:57.717463 2628 raft_consensus.cc:2243] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260204 02:53:57.717633 2628 raft_consensus.cc:2272] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:57.718649 2628 ts_tablet_manager.cc:1916] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fde467f2b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fde467ee7cc google::LogMessage::Flush() at ??:0
@ 0x7fde467eea11 google::LogMessage::~LogMessage() at ??:0
I20260204 02:53:57.724004 1271 tablet_service.cc:1552] Processing DeleteTablet for tablet 8bbcb21479124152831cf7d022fa4e5f with delete_type TABLET_DATA_DELETED (replaced by tablet ad418256b01c415aa306182f3eccd69b) from {username='slave'} at 127.0.0.1:39690
I20260204 02:53:57.724206 2628 ts_tablet_manager.cc:1929] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260204 02:53:57.724342 2628 log.cc:1199] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-0/wal/wals/8bbcb21479124152831cf7d022fa4e5f
I20260204 02:53:57.724774 2628 ts_tablet_manager.cc:1950] T 8bbcb21479124152831cf7d022fa4e5f P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Deleting consensus metadata
I20260204 02:53:57.729220 2542 catalog_manager.cc:6724] ReplaceTablet: tablet 8bbcb21479124152831cf7d022fa4e5f deleted and replaced by tablet ad418256b01c415aa306182f3eccd69b
I20260204 02:53:57.729825 2542 master_service.cc:954] ReplaceTablet: received request to replace tablet b321632c62d545d9b3ce83683af6de42 from {username='slave'} at 127.0.0.1:33206
I20260204 02:53:57.730389 2542 master_service.cc:954] ReplaceTablet: received request to replace tablet 1d7efc8694aa49e4ba8ab79721373607 from {username='slave'} at 127.0.0.1:33206
I20260204 02:53:57.730882 2542 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33206:
table { table_name: "default.Table_db7b7860261b41c692b36ca95b2d89e2" } new_table_name: "default.Table_667ea74f761c447c8538d777329b0b9f" modify_external_catalogs: true
I20260204 02:53:57.735280 2526 catalog_manager.cc:4985] TS 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): tablet 8bbcb21479124152831cf7d022fa4e5f (table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]) successfully deleted
I20260204 02:53:57.735554 2634 tablet_replica.cc:333] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5: stopping tablet replica
I20260204 02:53:57.735822 2634 raft_consensus.cc:2243] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [term 1 LEADER]: Raft consensus shutting down.
I20260204 02:53:57.735929 1136 tablet_service.cc:1552] Processing DeleteTablet for tablet 8bbcb21479124152831cf7d022fa4e5f with delete_type TABLET_DATA_DELETED (replaced by tablet ad418256b01c415aa306182f3eccd69b) from {username='slave'} at 127.0.0.1:41482
I20260204 02:53:57.736210 2634 raft_consensus.cc:2272] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:57.736794 2634 ts_tablet_manager.cc:1916] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:57.742461 1135 tablet_service.cc:1505] Processing CreateTablet for tablet a1f6cc066dad428ead30300716e58dfd (DEFAULT_TABLE table=default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:57.742894 1135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1f6cc066dad428ead30300716e58dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:57.743094 2634 ts_tablet_manager.cc:1929] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260204 02:53:57.743196 2634 log.cc:1199] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal/wals/8bbcb21479124152831cf7d022fa4e5f
I20260204 02:53:57.743619 2634 ts_tablet_manager.cc:1950] T 8bbcb21479124152831cf7d022fa4e5f P e2efdafb1246484a98dc5408be6746b5: Deleting consensus metadata
I20260204 02:53:57.744231 1273 tablet_service.cc:1505] Processing CreateTablet for tablet 38b39855b7dd4ea58a1cfc97849b088d (DEFAULT_TABLE table=default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:57.744551 1273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b39855b7dd4ea58a1cfc97849b088d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:57.744653 1272 tablet_service.cc:1505] Processing CreateTablet for tablet ad418256b01c415aa306182f3eccd69b (DEFAULT_TABLE table=default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:57.744920 1272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad418256b01c415aa306182f3eccd69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:57.748771 997 tablet_service.cc:1505] Processing CreateTablet for tablet a1f6cc066dad428ead30300716e58dfd (DEFAULT_TABLE table=default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:57.749146 997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1f6cc066dad428ead30300716e58dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:57.750910 2528 catalog_manager.cc:4985] TS e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): tablet 8bbcb21479124152831cf7d022fa4e5f (table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]) successfully deleted
I20260204 02:53:57.751328 1271 tablet_service.cc:1505] Processing CreateTablet for tablet a1f6cc066dad428ead30300716e58dfd (DEFAULT_TABLE table=default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:57.751668 1271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1f6cc066dad428ead30300716e58dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:57.752238 2639 tablet_bootstrap.cc:492] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:57.754060 2639 tablet_bootstrap.cc:654] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
@ 0x7fde5d2227f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260204 02:53:57.760380 2639 tablet_bootstrap.cc:492] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:57.760511 2639 ts_tablet_manager.cc:1403] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260204 02:53:57.760934 1138 tablet_service.cc:1505] Processing CreateTablet for tablet 38b39855b7dd4ea58a1cfc97849b088d (DEFAULT_TABLE table=default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:57.761307 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b39855b7dd4ea58a1cfc97849b088d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:57.761242 2639 raft_consensus.cc:359] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.761404 2639 raft_consensus.cc:385] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:57.761472 2639 raft_consensus.cc:740] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:57.761647 2641 tablet_bootstrap.cc:492] T a1f6cc066dad428ead30300716e58dfd P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
@ 0x7fde5d2210a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260204 02:53:57.763195 2639 consensus_queue.cc:260] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.763597 2641 tablet_bootstrap.cc:654] T a1f6cc066dad428ead30300716e58dfd P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:57.763840 2638 tablet_bootstrap.cc:492] T a1f6cc066dad428ead30300716e58dfd P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:57.765426 2635 tablet_replica.cc:333] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8: stopping tablet replica
I20260204 02:53:57.765614 2638 tablet_bootstrap.cc:654] T a1f6cc066dad428ead30300716e58dfd P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:57.765678 2635 raft_consensus.cc:2243] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Raft consensus shutting down.
@ 0x7fde5d228d1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260204 02:53:57.765904 2635 raft_consensus.cc:2272] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:53:57.766551 2635 ts_tablet_manager.cc:1916] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260204 02:53:57.768011 2639 ts_tablet_manager.cc:1434] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
@ 0x7fde5d228b99 std::_Function_handler<>::_M_invoke() at ??:0
I20260204 02:53:57.769924 998 tablet_service.cc:1505] Processing CreateTablet for tablet 38b39855b7dd4ea58a1cfc97849b088d (DEFAULT_TABLE table=default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:57.770025 2639 tablet_bootstrap.cc:492] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:57.770247 998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b39855b7dd4ea58a1cfc97849b088d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:57.771956 2639 tablet_bootstrap.cc:654] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7fde49563f99 kudu::Thread::SuperviseThread() at ??:0
I20260204 02:53:57.778735 1138 tablet_service.cc:1505] Processing CreateTablet for tablet ad418256b01c415aa306182f3eccd69b (DEFAULT_TABLE table=default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:57.778283 997 tablet_service.cc:1505] Processing CreateTablet for tablet ad418256b01c415aa306182f3eccd69b (DEFAULT_TABLE table=default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260204 02:53:57.779080 997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad418256b01c415aa306182f3eccd69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260204 02:53:57.779124 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad418256b01c415aa306182f3eccd69b. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fde4cb706db start_thread at ??:0
@ 0x7fde4569f71f clone at ??:0
I20260204 02:53:57.784044 2635 ts_tablet_manager.cc:1929] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260204 02:53:57.784180 2635 log.cc:1199] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-1/wal/wals/8bbcb21479124152831cf7d022fa4e5f
I20260204 02:53:57.784634 2635 ts_tablet_manager.cc:1950] T 8bbcb21479124152831cf7d022fa4e5f P 4dc2992e324049698330803c9ed2e3f8: Deleting consensus metadata
I20260204 02:53:57.786329 2639 tablet_bootstrap.cc:492] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:57.786491 2639 ts_tablet_manager.cc:1403] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.004s
I20260204 02:53:57.787370 2641 tablet_bootstrap.cc:492] T a1f6cc066dad428ead30300716e58dfd P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:57.787269 2639 raft_consensus.cc:359] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.787513 2639 raft_consensus.cc:385] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:57.787544 2641 ts_tablet_manager.cc:1403] T a1f6cc066dad428ead30300716e58dfd P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.004s
I20260204 02:53:57.787592 2639 raft_consensus.cc:740] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:57.787849 2639 consensus_queue.cc:260] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.788259 2641 raft_consensus.cc:359] T a1f6cc066dad428ead30300716e58dfd P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.788419 2641 raft_consensus.cc:385] T a1f6cc066dad428ead30300716e58dfd P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:57.788496 2641 raft_consensus.cc:740] T a1f6cc066dad428ead30300716e58dfd P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:57.788712 2641 consensus_queue.cc:260] T a1f6cc066dad428ead30300716e58dfd P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.789078 1064 heartbeater.cc:499] Master 127.0.104.188:38789 was elected leader, sending a full tablet report...
I20260204 02:53:57.790681 2641 ts_tablet_manager.cc:1434] T a1f6cc066dad428ead30300716e58dfd P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260204 02:53:57.790926 2641 tablet_bootstrap.cc:492] T 38b39855b7dd4ea58a1cfc97849b088d P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:57.792526 2639 ts_tablet_manager.cc:1434] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260204 02:53:57.792574 2641 tablet_bootstrap.cc:654] T 38b39855b7dd4ea58a1cfc97849b088d P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:57.792820 2639 tablet_bootstrap.cc:492] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5: Bootstrap starting.
I20260204 02:53:57.794348 2639 tablet_bootstrap.cc:654] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:57.794384 2638 tablet_bootstrap.cc:492] T a1f6cc066dad428ead30300716e58dfd P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:57.794553 2638 ts_tablet_manager.cc:1403] T a1f6cc066dad428ead30300716e58dfd P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.003s
I20260204 02:53:57.795343 2638 raft_consensus.cc:359] T a1f6cc066dad428ead30300716e58dfd P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.795558 2638 raft_consensus.cc:385] T a1f6cc066dad428ead30300716e58dfd P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:57.795667 2638 raft_consensus.cc:740] T a1f6cc066dad428ead30300716e58dfd P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:57.795903 2638 consensus_queue.cc:260] T a1f6cc066dad428ead30300716e58dfd P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.796844 1203 heartbeater.cc:499] Master 127.0.104.188:38789 was elected leader, sending a full tablet report...
I20260204 02:53:57.800004 2638 ts_tablet_manager.cc:1434] T a1f6cc066dad428ead30300716e58dfd P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260204 02:53:57.800925 2638 tablet_bootstrap.cc:492] T 38b39855b7dd4ea58a1cfc97849b088d P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:57.802467 2638 tablet_bootstrap.cc:654] T 38b39855b7dd4ea58a1cfc97849b088d P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:57.804328 2641 tablet_bootstrap.cc:492] T 38b39855b7dd4ea58a1cfc97849b088d P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:57.804479 2641 ts_tablet_manager.cc:1403] T 38b39855b7dd4ea58a1cfc97849b088d P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20260204 02:53:57.805773 2641 raft_consensus.cc:359] T 38b39855b7dd4ea58a1cfc97849b088d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.805964 2641 raft_consensus.cc:385] T 38b39855b7dd4ea58a1cfc97849b088d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:57.806056 2641 raft_consensus.cc:740] T 38b39855b7dd4ea58a1cfc97849b088d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:57.806293 2638 tablet_bootstrap.cc:492] T 38b39855b7dd4ea58a1cfc97849b088d P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:57.806428 2638 ts_tablet_manager.cc:1403] T 38b39855b7dd4ea58a1cfc97849b088d P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260204 02:53:57.806331 2641 consensus_queue.cc:260] T 38b39855b7dd4ea58a1cfc97849b088d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.806834 2641 ts_tablet_manager.cc:1434] T 38b39855b7dd4ea58a1cfc97849b088d P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260204 02:53:57.807386 2641 tablet_bootstrap.cc:492] T ad418256b01c415aa306182f3eccd69b P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Bootstrap starting.
I20260204 02:53:57.807942 2639 tablet_bootstrap.cc:492] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5: No bootstrap required, opened a new log
I20260204 02:53:57.808980 2639 ts_tablet_manager.cc:1403] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.013s
I20260204 02:53:57.809027 2641 tablet_bootstrap.cc:654] T ad418256b01c415aa306182f3eccd69b P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:57.810281 2639 raft_consensus.cc:359] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.810436 2639 raft_consensus.cc:385] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:57.810432 2638 raft_consensus.cc:359] T 38b39855b7dd4ea58a1cfc97849b088d P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.810514 2639 raft_consensus.cc:740] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Initialized, Role: FOLLOWER
I20260204 02:53:57.810608 2638 raft_consensus.cc:385] T 38b39855b7dd4ea58a1cfc97849b088d P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:57.810684 2638 raft_consensus.cc:740] T 38b39855b7dd4ea58a1cfc97849b088d P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:57.810760 2639 consensus_queue.cc:260] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.810926 2638 consensus_queue.cc:260] T 38b39855b7dd4ea58a1cfc97849b088d P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.811429 2638 ts_tablet_manager.cc:1434] T 38b39855b7dd4ea58a1cfc97849b088d P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260204 02:53:57.811740 2638 tablet_bootstrap.cc:492] T ad418256b01c415aa306182f3eccd69b P 4dc2992e324049698330803c9ed2e3f8: Bootstrap starting.
I20260204 02:53:57.813378 2638 tablet_bootstrap.cc:654] T ad418256b01c415aa306182f3eccd69b P 4dc2992e324049698330803c9ed2e3f8: Neither blocks nor log segments found. Creating new log.
I20260204 02:53:57.815682 2639 ts_tablet_manager.cc:1434] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260204 02:53:57.818096 2641 tablet_bootstrap.cc:492] T ad418256b01c415aa306182f3eccd69b P 8d7605b7e0f949b3ba7ffaddd1ca8db3: No bootstrap required, opened a new log
I20260204 02:53:57.818830 2641 ts_tablet_manager.cc:1403] T ad418256b01c415aa306182f3eccd69b P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260204 02:53:57.819752 2641 raft_consensus.cc:359] T ad418256b01c415aa306182f3eccd69b P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.819932 2641 raft_consensus.cc:385] T ad418256b01c415aa306182f3eccd69b P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:57.820003 2641 raft_consensus.cc:740] T ad418256b01c415aa306182f3eccd69b P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7605b7e0f949b3ba7ffaddd1ca8db3, State: Initialized, Role: FOLLOWER
I20260204 02:53:57.820235 2641 consensus_queue.cc:260] T ad418256b01c415aa306182f3eccd69b P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.821457 2641 ts_tablet_manager.cc:1434] T ad418256b01c415aa306182f3eccd69b P 8d7605b7e0f949b3ba7ffaddd1ca8db3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260204 02:53:57.823975 2638 tablet_bootstrap.cc:492] T ad418256b01c415aa306182f3eccd69b P 4dc2992e324049698330803c9ed2e3f8: No bootstrap required, opened a new log
I20260204 02:53:57.824126 2638 ts_tablet_manager.cc:1403] T ad418256b01c415aa306182f3eccd69b P 4dc2992e324049698330803c9ed2e3f8: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20260204 02:53:57.824993 2638 raft_consensus.cc:359] T ad418256b01c415aa306182f3eccd69b P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.825122 2638 raft_consensus.cc:385] T ad418256b01c415aa306182f3eccd69b P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260204 02:53:57.825181 2638 raft_consensus.cc:740] T ad418256b01c415aa306182f3eccd69b P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc2992e324049698330803c9ed2e3f8, State: Initialized, Role: FOLLOWER
I20260204 02:53:57.825403 2638 consensus_queue.cc:260] T ad418256b01c415aa306182f3eccd69b P 4dc2992e324049698330803c9ed2e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.826013 2638 ts_tablet_manager.cc:1434] T ad418256b01c415aa306182f3eccd69b P 4dc2992e324049698330803c9ed2e3f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260204 02:53:57.830978 2485 raft_consensus.cc:493] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:57.831111 2485 raft_consensus.cc:515] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.832021 2485 leader_election.cc:290] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571), 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:57.832724 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38b39855b7dd4ea58a1cfc97849b088d" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:57.832942 1019 raft_consensus.cc:2468] T 38b39855b7dd4ea58a1cfc97849b088d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 0.
I20260204 02:53:57.832894 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38b39855b7dd4ea58a1cfc97849b088d" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8" is_pre_election: true
I20260204 02:53:57.833112 1158 raft_consensus.cc:2468] T 38b39855b7dd4ea58a1cfc97849b088d P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 0.
I20260204 02:53:57.833412 1227 leader_election.cc:304] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:57.833724 2485 raft_consensus.cc:2804] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:57.833843 2485 raft_consensus.cc:493] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:57.833927 2485 raft_consensus.cc:3060] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:57.836716 2485 raft_consensus.cc:515] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.837723 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38b39855b7dd4ea58a1cfc97849b088d" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8"
I20260204 02:53:57.837828 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38b39855b7dd4ea58a1cfc97849b088d" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
I20260204 02:53:57.837901 1158 raft_consensus.cc:3060] T 38b39855b7dd4ea58a1cfc97849b088d P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:57.837998 1019 raft_consensus.cc:3060] T 38b39855b7dd4ea58a1cfc97849b088d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:57.839815 1019 raft_consensus.cc:2468] T 38b39855b7dd4ea58a1cfc97849b088d P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 1.
I20260204 02:53:57.837340 2485 leader_election.cc:290] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 election: Requested vote from peers 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571), 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:57.840279 1227 leader_election.cc:304] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:57.840531 2485 raft_consensus.cc:2804] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:57.840582 1158 raft_consensus.cc:2468] T 38b39855b7dd4ea58a1cfc97849b088d P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 1.
I20260204 02:53:57.840946 2485 raft_consensus.cc:697] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [term 1 LEADER]: Becoming Leader. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Running, Role: LEADER
I20260204 02:53:57.841238 2485 consensus_queue.cc:237] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.886788 2485 raft_consensus.cc:493] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:57.887032 2485 raft_consensus.cc:515] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.887743 2485 leader_election.cc:290] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571), 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:57.888319 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad418256b01c415aa306182f3eccd69b" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:57.888329 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad418256b01c415aa306182f3eccd69b" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8" is_pre_election: true
I20260204 02:53:57.888528 1019 raft_consensus.cc:2468] T ad418256b01c415aa306182f3eccd69b P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 0.
I20260204 02:53:57.888537 1158 raft_consensus.cc:2468] T ad418256b01c415aa306182f3eccd69b P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 0.
I20260204 02:53:57.888999 1227 leader_election.cc:304] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:57.889321 2485 raft_consensus.cc:2804] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:57.889653 2485 raft_consensus.cc:493] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:57.889812 2485 raft_consensus.cc:3060] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:57.892438 2485 raft_consensus.cc:515] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.893079 2485 leader_election.cc:290] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 election: Requested vote from peers 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571), 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:57.893415 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad418256b01c415aa306182f3eccd69b" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8"
I20260204 02:53:57.893464 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad418256b01c415aa306182f3eccd69b" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
I20260204 02:53:57.893589 1158 raft_consensus.cc:3060] T ad418256b01c415aa306182f3eccd69b P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:57.893631 1019 raft_consensus.cc:3060] T ad418256b01c415aa306182f3eccd69b P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:57.895428 1019 raft_consensus.cc:2468] T ad418256b01c415aa306182f3eccd69b P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 1.
I20260204 02:53:57.895906 1227 leader_election.cc:304] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:57.896142 2485 raft_consensus.cc:2804] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:57.896328 2485 raft_consensus.cc:697] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [term 1 LEADER]: Becoming Leader. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Running, Role: LEADER
I20260204 02:53:57.896628 2485 consensus_queue.cc:237] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:57.897730 1158 raft_consensus.cc:2468] T ad418256b01c415aa306182f3eccd69b P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 1.
W20260204 02:53:57.976438 1094 connection.cc:537] server connection from 127.0.0.1:41482 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:53:57.976495 952 connection.cc:537] server connection from 127.0.0.1:56548 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:53:57.976672 1346 connection.cc:537] client connection to 127.0.104.188:38789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:53:57.976752 952 connection.cc:537] client connection to 127.0.104.188:38789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:53:57.977378 1346 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.188:38789)
W20260204 02:53:57.978116 1346 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.188:38789)
W20260204 02:53:57.978281 1091 connection.cc:537] client connection to 127.0.104.188:38789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:53:57.978513 1346 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.188:38789)
W20260204 02:53:57.979204 1346 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.188:38789)
I20260204 02:53:58.123760 2485 raft_consensus.cc:493] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:53:58.123960 2485 raft_consensus.cc:515] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:58.124502 2485 leader_election.cc:290] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571), 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:58.124985 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1f6cc066dad428ead30300716e58dfd" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" is_pre_election: true
I20260204 02:53:58.124955 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1f6cc066dad428ead30300716e58dfd" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8" is_pre_election: true
I20260204 02:53:58.125186 1019 raft_consensus.cc:2468] T a1f6cc066dad428ead30300716e58dfd P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 0.
I20260204 02:53:58.125186 1158 raft_consensus.cc:2468] T a1f6cc066dad428ead30300716e58dfd P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 0.
I20260204 02:53:58.125552 1227 leader_election.cc:304] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:58.125782 2485 raft_consensus.cc:2804] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260204 02:53:58.125896 2485 raft_consensus.cc:493] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:53:58.125979 2485 raft_consensus.cc:3060] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:58.128732 2485 raft_consensus.cc:515] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:58.129312 2485 leader_election.cc:290] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 election: Requested vote from peers 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571), 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:53:58.129796 1019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1f6cc066dad428ead30300716e58dfd" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3"
I20260204 02:53:58.129980 1019 raft_consensus.cc:3060] T a1f6cc066dad428ead30300716e58dfd P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:58.132577 1019 raft_consensus.cc:2468] T a1f6cc066dad428ead30300716e58dfd P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 1.
I20260204 02:53:58.132946 1227 leader_election.cc:304] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5; no voters:
I20260204 02:53:58.133147 1158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1f6cc066dad428ead30300716e58dfd" candidate_uuid: "e2efdafb1246484a98dc5408be6746b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc2992e324049698330803c9ed2e3f8"
I20260204 02:53:58.133306 2485 raft_consensus.cc:2804] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260204 02:53:58.133340 1158 raft_consensus.cc:3060] T a1f6cc066dad428ead30300716e58dfd P 4dc2992e324049698330803c9ed2e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260204 02:53:58.133502 2485 raft_consensus.cc:697] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [term 1 LEADER]: Becoming Leader. State: Replica: e2efdafb1246484a98dc5408be6746b5, State: Running, Role: LEADER
I20260204 02:53:58.133914 2485 consensus_queue.cc:237] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } }
I20260204 02:53:58.136051 1158 raft_consensus.cc:2468] T a1f6cc066dad428ead30300716e58dfd P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2efdafb1246484a98dc5408be6746b5 in term 1.
I20260204 02:53:58.241916 2666 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 202e4a0debb24c8aa89a4724d72fcf37)
I20260204 02:53:58.242077 2666 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:58.242722 2666 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:58.244773 2206 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
W20260204 02:53:58.244930 2208 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:53:58.245085 2208 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:58.245289 2666 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260204 02:53:58.601125 2485 consensus_queue.cc:1048] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:58.606554 2657 consensus_queue.cc:1048] T 38b39855b7dd4ea58a1cfc97849b088d P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:58.726284 2676 consensus_queue.cc:1048] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:58.731730 2676 consensus_queue.cc:1048] T ad418256b01c415aa306182f3eccd69b P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:58.950224 2682 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 202e4a0debb24c8aa89a4724d72fcf37)
I20260204 02:53:58.950389 2682 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:53:58.950953 2682 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:53:58.952788 2208 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:53:58.952968 2206 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:53:58.953078 2206 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:53:58.953277 2682 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260204 02:53:58.984617 2676 consensus_queue.cc:1048] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:58.991276 2657 consensus_queue.cc:1048] T a1f6cc066dad428ead30300716e58dfd P e2efdafb1246484a98dc5408be6746b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260204 02:53:59.004640 418 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-task6hEvLx/build/asan/bin/kudu pid 2188
I20260204 02:53:59.013195 2254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260204 02:53:59.122159 418 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6hEvLx/build/asan/bin/kudu with pid 2188
I20260204 02:53:59.513237 2692 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:53:59.513425 2692 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:53:59.514195 2692 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:53:59.515035 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:53:59.515244 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:53:59.515376 1091 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:53:59.515645 2692 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:53:59.649609 2695 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:53:59.649807 2695 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:53:59.650422 2695 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:53:59.651119 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:53:59.651284 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:53:59.651443 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:53:59.651723 2695 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:53:59.958691 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.189:46765
--webserver_interface=127.0.104.189
--webserver_port=36571
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260204 02:54:00.146795 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 11: this message will repeat every 5th retry.
W20260204 02:54:00.197281 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 11: this message will repeat every 5th retry.
W20260204 02:54:00.276073 2704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:54:00.276379 2704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:54:00.276521 2704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260204 02:54:00.283912 2704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260204 02:54:00.283988 2704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:54:00.284066 2704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260204 02:54:00.284134 2704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260204 02:54:00.293495 2704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.189:46765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.189
--webserver_port=36571
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:54:00.295028 2704 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:54:00.296814 2704 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260204 02:54:00.304556 2714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:54:00.304591 2715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:54:00.304884 2717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:54:00.305475 2704 server_base.cc:1060] running on GCE node
I20260204 02:54:00.306299 2704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:54:00.307605 2704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:54:00.308810 2704 hybrid_clock.cc:648] HybridClock initialized: now 1770173640308762 us; error 49 us; skew 500 ppm
I20260204 02:54:00.309129 2704 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:54:00.311198 2704 webserver.cc:492] Webserver started at http://127.0.104.189:36571/ using document root <none> and password file <none>
I20260204 02:54:00.311919 2704 fs_manager.cc:362] Metadata directory not provided
I20260204 02:54:00.312031 2704 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:54:00.316838 2704 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260204 02:54:00.320068 2723 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:54:00.321298 2704 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260204 02:54:00.321466 2704 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
uuid: "78eb37be156448e8879b0df55832ab2d"
format_stamp: "Formatted at 2026-02-04 02:53:30 on dist-test-slave-xxhz"
I20260204 02:54:00.322088 2704 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:54:00.341104 2704 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:54:00.341853 2704 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:54:00.342109 2704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260204 02:54:00.360635 2704 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.189:46765
I20260204 02:54:00.360664 2774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.189:46765 every 8 connection(s)
I20260204 02:54:00.362126 2704 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-1/data/info.pb
I20260204 02:54:00.365547 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 2704
I20260204 02:54:00.369383 2775 sys_catalog.cc:263] Verifying existing consensus state
I20260204 02:54:00.373718 2775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Bootstrap starting.
I20260204 02:54:00.386689 2775 log.cc:826] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Log is configured to *not* fsync() on all Append() calls
I20260204 02:54:00.419903 2775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=49 overwritten=0 applied=49 ignored=0} inserts{seen=31 ignored=0} mutations{seen=38 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260204 02:54:00.420415 2775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Bootstrap complete.
I20260204 02:54:00.425140 2775 raft_consensus.cc:359] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:00.425799 2775 raft_consensus.cc:740] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78eb37be156448e8879b0df55832ab2d, State: Initialized, Role: FOLLOWER
I20260204 02:54:00.426353 2775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 49, Last appended: 8.49, Last appended by leader: 49, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:00.427207 2780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:54:00.427449 2780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:54:00.428220 2775 sys_catalog.cc:565] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: configured and running, proceeding with master startup.
I20260204 02:54:00.441280 2794 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:54:00.441479 2794 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: loading cluster ID for follower catalog manager: success
I20260204 02:54:00.437305 2704 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:54:00.443213 2704 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:54:00.446260 2794 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: acquiring CA information for follower catalog manager: success
I20260204 02:54:00.449194 2794 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260204 02:54:00.653072 1205 heartbeater.cc:344] Connected to a master server at 127.0.104.189:46765
I20260204 02:54:00.655869 2739 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" instance_seqno: 1770173612604702) as {username='slave'} at 127.0.104.130:59299; Asking this server to re-register.
I20260204 02:54:00.656548 1205 heartbeater.cc:461] Registering TS with master...
I20260204 02:54:00.658201 2739 ts_manager.cc:194] Registered new tserver with Master: 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:54:00.714090 1066 heartbeater.cc:344] Connected to a master server at 127.0.104.189:46765
I20260204 02:54:00.715901 2739 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" instance_seqno: 1770173612117836) as {username='slave'} at 127.0.104.129:59029; Asking this server to re-register.
I20260204 02:54:00.716497 1066 heartbeater.cc:461] Registering TS with master...
I20260204 02:54:00.717919 2739 ts_manager.cc:194] Registered new tserver with Master: 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:54:00.725477 1341 heartbeater.cc:344] Connected to a master server at 127.0.104.189:46765
I20260204 02:54:00.727222 2739 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" instance_seqno: 1770173613100203) as {username='slave'} at 127.0.104.131:41383; Asking this server to re-register.
I20260204 02:54:00.727880 1341 heartbeater.cc:461] Registering TS with master...
I20260204 02:54:00.729139 2739 ts_manager.cc:194] Registered new tserver with Master: e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:54:00.907532 2780 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:54:00.907792 2780 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:00.909416 2780 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:54:00.910141 2724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
W20260204 02:54:00.911334 2726 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:54:00.911545 2724 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:54:00.911707 2724 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:54:00.912026 2780 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260204 02:54:01.553013 2813 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:54:01.553157 2813 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:01.553925 2813 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:54:01.555646 2726 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:54:01.555816 2724 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:54:01.555956 2724 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:54:01.556180 2813 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260204 02:54:02.624933 2830 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:54:02.625160 2830 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:02.625811 2830 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:54:02.628618 2726 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:54:02.628921 2724 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789): Network error: Client connection negotiation failed: client connection to 127.0.104.188:38789: connect: Connection refused (error 111)
I20260204 02:54:02.629040 2724 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78eb37be156448e8879b0df55832ab2d; no voters: 202e4a0debb24c8aa89a4724d72fcf37, 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:54:02.629333 2830 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260204 02:54:03.437263 418 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6hEvLx/build/asan/bin/kudu
/tmp/dist-test-task6hEvLx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.188:38789
--webserver_interface=127.0.104.188
--webserver_port=40055
--builtin_ntp_servers=127.0.104.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260204 02:54:03.750741 2847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260204 02:54:03.751108 2847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260204 02:54:03.751231 2847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260204 02:54:03.759388 2847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260204 02:54:03.759529 2847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260204 02:54:03.759589 2847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260204 02:54:03.759671 2847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260204 02:54:03.769698 2847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:36271
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.188:38789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.188
--webserver_port=40055
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type FASTDEBUG
built by None at 04 Feb 2026 02:43:24 UTC on 24a791456cd2
build id 10336
ASAN enabled
I20260204 02:54:03.771574 2847 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260204 02:54:03.773300 2847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260204 02:54:03.781798 2858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:54:03.781826 2861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260204 02:54:03.781862 2859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260204 02:54:03.782727 2847 server_base.cc:1060] running on GCE node
I20260204 02:54:03.783389 2847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260204 02:54:03.784680 2847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260204 02:54:03.786005 2847 hybrid_clock.cc:648] HybridClock initialized: now 1770173643785964 us; error 42 us; skew 500 ppm
I20260204 02:54:03.786386 2847 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260204 02:54:03.788421 2847 webserver.cc:492] Webserver started at http://127.0.104.188:40055/ using document root <none> and password file <none>
I20260204 02:54:03.789157 2847 fs_manager.cc:362] Metadata directory not provided
I20260204 02:54:03.789306 2847 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260204 02:54:03.794678 2847 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260204 02:54:03.797556 2867 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260204 02:54:03.798839 2847 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260204 02:54:03.799001 2847 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data,/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
uuid: "202e4a0debb24c8aa89a4724d72fcf37"
format_stamp: "Formatted at 2026-02-04 02:53:31 on dist-test-slave-xxhz"
I20260204 02:54:03.799748 2847 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260204 02:54:03.837356 2847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260204 02:54:03.838232 2847 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260204 02:54:03.838497 2847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260204 02:54:03.858268 2847 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.188:38789
I20260204 02:54:03.858284 2918 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.188:38789 every 8 connection(s)
I20260204 02:54:03.859990 2847 server_base.cc:1192] Dumped server information to /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/master-2/data/info.pb
I20260204 02:54:03.866609 2919 sys_catalog.cc:263] Verifying existing consensus state
I20260204 02:54:03.867090 418 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6hEvLx/build/asan/bin/kudu as pid 2847
I20260204 02:54:03.871114 2919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap starting.
I20260204 02:54:03.884235 2919 log.cc:826] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Log is configured to *not* fsync() on all Append() calls
I20260204 02:54:03.920750 2919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap replayed 1/1 log segments. Stats: ops{read=49 overwritten=0 applied=49 ignored=0} inserts{seen=31 ignored=0} mutations{seen=38 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260204 02:54:03.921319 2919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: Bootstrap complete.
I20260204 02:54:03.926338 2919 raft_consensus.cc:359] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:03.927016 2919 raft_consensus.cc:740] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202e4a0debb24c8aa89a4724d72fcf37, State: Initialized, Role: FOLLOWER
I20260204 02:54:03.927667 2919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 49, Last appended: 8.49, Last appended by leader: 49, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:03.928373 2923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:54:03.928630 2923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:54:03.929538 2919 sys_catalog.cc:565] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: configured and running, proceeding with master startup.
I20260204 02:54:03.944777 2937 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:54:03.944964 2937 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: loading cluster ID for follower catalog manager: success
I20260204 02:54:03.943351 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:54:03.949594 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:54:03.949787 2937 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: acquiring CA information for follower catalog manager: success
I20260204 02:54:03.953172 2937 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260204 02:54:03.955766 2941 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260204 02:54:03.956048 2941 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:03.959797 2941 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
W20260204 02:54:03.963274 2726 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:54:03.976397 2894 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78eb37be156448e8879b0df55832ab2d" candidate_term: 9 candidate_status { last_received { term: 8 index: 49 } } ignore_live_leader: false dest_uuid: "202e4a0debb24c8aa89a4724d72fcf37" is_pre_election: true
I20260204 02:54:03.977149 2894 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78eb37be156448e8879b0df55832ab2d in term 8.
I20260204 02:54:03.978204 2724 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:54:03.978786 2941 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260204 02:54:03.978919 2941 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260204 02:54:03.979023 2941 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 8 FOLLOWER]: Advancing to term 9
I20260204 02:54:03.981714 1064 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:54:03.981866 1064 heartbeater.cc:499] Master 127.0.104.188:38789 was elected leader, sending a full tablet report...
I20260204 02:54:03.984156 2941 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:03.984865 2941 leader_election.cc:290] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 election: Requested vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 202e4a0debb24c8aa89a4724d72fcf37 (127.0.104.188:38789)
I20260204 02:54:03.985357 2894 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78eb37be156448e8879b0df55832ab2d" candidate_term: 9 candidate_status { last_received { term: 8 index: 49 } } ignore_live_leader: false dest_uuid: "202e4a0debb24c8aa89a4724d72fcf37"
I20260204 02:54:03.985574 2894 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 8 FOLLOWER]: Advancing to term 9
W20260204 02:54:03.986183 2726 leader_election.cc:336] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:54:03.986291 2883 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" instance_seqno: 1770173612117836) as {username='slave'} at 127.0.104.129:54821; Asking this server to re-register.
I20260204 02:54:03.987353 1064 heartbeater.cc:461] Registering TS with master...
I20260204 02:54:03.990464 2894 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78eb37be156448e8879b0df55832ab2d in term 9.
I20260204 02:54:03.990943 2724 leader_election.cc:304] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37, 78eb37be156448e8879b0df55832ab2d; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff
I20260204 02:54:03.991315 2941 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 9 FOLLOWER]: Leader election won for term 9
I20260204 02:54:03.991842 2941 raft_consensus.cc:697] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [term 9 LEADER]: Becoming Leader. State: Replica: 78eb37be156448e8879b0df55832ab2d, State: Running, Role: LEADER
I20260204 02:54:03.992223 2883 ts_manager.cc:194] Registered new tserver with Master: 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767)
I20260204 02:54:03.992446 2941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 8.49, Last appended by leader: 49, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:03.994195 2846 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:54:03.994372 2846 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:54:03.995042 2846 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:54:03.995958 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:54:03.996215 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:54:03.996359 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:03.996162 2949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78eb37be156448e8879b0df55832ab2d. Latest consensus state: current_term: 9 leader_uuid: "78eb37be156448e8879b0df55832ab2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:54:03.996490 2949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: LEADER
I20260204 02:54:03.996613 2846 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:54:03.997216 2951 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260204 02:54:03.999677 2951 catalog_manager.cc:679] Loaded metadata for table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76]
I20260204 02:54:04.000129 2951 catalog_manager.cc:679] Loaded metadata for table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b]
I20260204 02:54:04.000455 2951 catalog_manager.cc:679] Loaded metadata for table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793]
I20260204 02:54:04.000779 2951 catalog_manager.cc:679] Loaded metadata for table default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796]
I20260204 02:54:04.001072 2951 catalog_manager.cc:679] Loaded metadata for table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2]
I20260204 02:54:04.001423 2951 catalog_manager.cc:679] Loaded metadata for table default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2]
I20260204 02:54:04.001732 2951 catalog_manager.cc:679] Loaded metadata for table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29]
I20260204 02:54:04.002075 2951 catalog_manager.cc:679] Loaded metadata for table default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5]
I20260204 02:54:04.002399 2951 catalog_manager.cc:679] Loaded metadata for table default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f]
I20260204 02:54:04.004225 2951 tablet_loader.cc:96] loaded metadata for tablet 01792fc91bba4ba891812bd25e006268 (table default.table_db7b7860261b41c692b36ca95b2d89e2 [id=8dded49817604d999f92e9157655e2f2])
I20260204 02:54:04.004601 2951 tablet_loader.cc:96] loaded metadata for tablet 15b2b6a7198c4ebb98daf4de751911a6 (table default.table_3a403dbfa3044e55abc223b3acd75b6b [id=333921075666492fb4e06fa35e04487b])
I20260204 02:54:04.005188 2951 tablet_loader.cc:96] loaded metadata for tablet 2b883ef926504d489f3a1c0aec6db438 (table default.table_a16e01078d0342e7abbb4ec634032416 [id=ceb020292ca6471fb47601b65b5ef0d5])
I20260204 02:54:04.005479 2951 tablet_loader.cc:96] loaded metadata for tablet 3758b5156dd04de984cd9313459d7cfb (table default.table_cb3f03c12e314422b462640be08ec293 [id=b27e50e8b8b84fd2972f1b7c8768dd29])
I20260204 02:54:04.005745 2951 tablet_loader.cc:96] loaded metadata for tablet 38b39855b7dd4ea58a1cfc97849b088d (table default.table_842bb3fe3b2944498784e63317dd1907 [id=22158e83a9534758a87a9a1f10eaba76])
I20260204 02:54:04.006714 2951 tablet_loader.cc:96] loaded metadata for tablet 8f1bc638fe8e48189c15d18209efdd1d (table default.table_69c350f50a63410c9da5a5e7b0862cf6 [id=5705cdf41ef94c8d94f3b3db54323796])
I20260204 02:54:04.007180 2951 tablet_loader.cc:96] loaded metadata for tablet a1f6cc066dad428ead30300716e58dfd (table default.table_3d3a816d407e429ea458b787f879ede0 [id=d8f5b0e4ab40417ba4f4b805c6c6941f])
I20260204 02:54:04.007457 2951 tablet_loader.cc:96] loaded metadata for tablet ad418256b01c415aa306182f3eccd69b (table default.table_79ae542654da4da5aba4d8c38d6abae7 [id=4b389805ed274ba89b2f4fa10039d793])
I20260204 02:54:04.007946 2951 tablet_loader.cc:96] loaded metadata for tablet b8094eec37cd40b9b1f4931f24d824a8 (table default.table_f734190e49874f9fb2c8dee62aa9e70b [id=7a55ab36a42842a6a7b1f572501cd1f2])
I20260204 02:54:04.008523 2951 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260204 02:54:04.009495 2951 catalog_manager.cc:1269] Loaded cluster ID: 0d852ac7150c4351a6008ce105768b85
I20260204 02:54:04.009588 2951 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260204 02:54:04.010784 2951 catalog_manager.cc:1514] Loading token signing keys...
I20260204 02:54:04.011786 2951 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Loaded TSK: 0
I20260204 02:54:04.012594 2951 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260204 02:54:04.013443 2951 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260204 02:54:04.014345 2951 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260204 02:54:04.035055 2739 master_service.cc:954] ReplaceTablet: received request to replace tablet 15b2b6a7198c4ebb98daf4de751911a6 from {username='slave'} at 127.0.0.1:39278
I20260204 02:54:04.036535 2732 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39278:
table { table_name: "default.Table_69c350f50a63410c9da5a5e7b0862cf6" } modify_external_catalogs: true
I20260204 02:54:04.036844 2732 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39278:
table { table_name: "default.Table_69c350f50a63410c9da5a5e7b0862cf6" } modify_external_catalogs: true
I20260204 02:54:04.037722 2733 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39278:
name: "default.Table_6bba790402724856976fb9e10e3c48fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:54:04.038599 2734 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39278:
name: "default.Table_e971e55de1b44441b34f0913c0d665a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:54:04.039198 2735 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39278:
name: "default.Table_3f77bbc9c07244dcb50a3b82a1fc8f1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:54:04.039677 2736 master_service.cc:954] ReplaceTablet: received request to replace tablet 01792fc91bba4ba891812bd25e006268 from {username='slave'} at 127.0.0.1:39278
I20260204 02:54:04.041087 2737 master_service.cc:954] ReplaceTablet: received request to replace tablet 3758b5156dd04de984cd9313459d7cfb from {username='slave'} at 127.0.0.1:39278
I20260204 02:54:04.042599 2738 master_service.cc:954] ReplaceTablet: received request to replace tablet 1d7efc8694aa49e4ba8ab79721373607 from {username='slave'} at 127.0.0.1:39278
I20260204 02:54:04.037719 2731 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39278:
name: "default.Table_c26f0462e2664a018e18bf0ef613c017"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:54:04.043453 2738 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39278:
name: "default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:54:04.043591 2740 master_service.cc:954] ReplaceTablet: received request to replace tablet 15b2b6a7198c4ebb98daf4de751911a6 from {username='slave'} at 127.0.0.1:39278
F20260204 02:54:04.046478 2793 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_3a403dbfa3044e55abc223b3acd75b6b, message={"server":"","servicePrincipal":"","db":"default","table":"table_3a403dbfa3044e55abc223b3acd75b6b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_3a403dbfa3044e55abc223b3acd75b6b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_3a403dbfa3044e55abc223b3acd75b6b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_3a403dbfa3044e55abc223b3acd75b6b\",\"kudu.table_id\":\"333921075666492fb4e06fa35e04487b\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1770173613, eventType=CREATE_TABLE, dbName=default, tableName=table_0312705ce5e140d4bcf88a92cab2bb7b, message={"server":"","servicePrincipal":"","db":"default","table":"table_0312705ce5e140d4bcf88a92cab2bb7b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1770173613},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/warehouse/table_0312705ce5e140d4bcf88a92cab2bb7b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0312705ce5e140d4bcf88a92cab2bb7b\",\"kudu.table_id\":\"8df7ca3e2276436e8bdb4480fc3dd711\",\"transient_lastDdlTime\":\"1770173613\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0d852ac7150c4351a6008ce105768b85\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1770173613,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f7afdb32dcd google::LogMessage::Fail() at ??:0
I20260204 02:54:04.049335 2894 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Refusing update from remote peer 78eb37be156448e8879b0df55832ab2d: Log matching property violated. Preceding OpId in replica: term: 8 index: 49. Preceding OpId from leader: term: 9 index: 51. (index mismatch)
W20260204 02:54:04.049892 2726 consensus_peers.cc:597] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d -> Peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Couldn't send request to peer 3e85e1dc6a5d4568ad365ba1d77398ff. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260204 02:54:04.050251 2949 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
@ 0x7f7afdb36b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f7afdb327cc google::LogMessage::Flush() at ??:0
@ 0x7f7afdb32a11 google::LogMessage::~LogMessage() at ??:0
I20260204 02:54:04.060261 2923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78eb37be156448e8879b0df55832ab2d. Latest consensus state: current_term: 9 leader_uuid: "78eb37be156448e8879b0df55832ab2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:54:04.060529 2923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f7b145667f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260204 02:54:04.062697 2952 mvcc.cc:204] Tried to move back new op lower bound from 7250631246016618496 to 7250631245797339136. Current Snapshot: MvccSnapshot[applied={T|T < 7250631246016618496}]
@ 0x7f7b145650a1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260204 02:54:04.070546 2949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "78eb37be156448e8879b0df55832ab2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:54:04.070785 2949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d [sys.catalog]: This master's current role is: LEADER
I20260204 02:54:04.073028 2740 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39278:
name: "default.Table_237c40ddb38c410c9ce533b72060712f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260204 02:54:04.073562 2923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "78eb37be156448e8879b0df55832ab2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } } }
I20260204 02:54:04.073745 2923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [sys.catalog]: This master's current role is: FOLLOWER
I20260204 02:54:04.073911 2739 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Sending DeleteTablet for 3 replicas of tablet 15b2b6a7198c4ebb98daf4de751911a6
@ 0x7f7b1456cd1c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f7b1456cb99 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
I20260204 02:54:04.088899 2736 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 78eb37be156448e8879b0df55832ab2d: Sending DeleteTablet for 3 replicas of tablet 01792fc91bba4ba891812bd25e006268
I20260204 02:54:04.089357 2736 catalog_manager.cc:6724] ReplaceTablet: tablet 01792fc91bba4ba891812bd25e006268 deleted and replaced by tablet 93927235c4014ddabc37a56b7b265f79
I20260204 02:54:04.092085 2739 catalog_manager.cc:6724] ReplaceTablet: tablet 15b2b6a7198c4ebb98daf4de751911a6 deleted and replaced by tablet 36195a94f1974dabaa6f5f4763644808
@ 0x7f7b008a7f99 kudu::Thread::SuperviseThread() at ??:0
I20260204 02:54:04.099960 2736 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39278:
table { table_name: "default.Table_79ae542654da4da5aba4d8c38d6abae7" } modify_external_catalogs: true
I20260204 02:54:04.100308 2736 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39278:
table { table_name: "default.Table_79ae542654da4da5aba4d8c38d6abae7" } modify_external_catalogs: true
I20260204 02:54:04.100751 2739 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39278:
name: "default.Table_a2c8b05d2d724f50aaf92bddb8a266e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f7b03eb46db start_thread at ??:0
@ 0x7f7afc9e371f clone at ??:0
I20260204 02:54:04.108640 1271 tablet_service.cc:1552] Processing DeleteTablet for tablet 15b2b6a7198c4ebb98daf4de751911a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 36195a94f1974dabaa6f5f4763644808) from {username='slave'} at 127.0.0.1:39700
I20260204 02:54:04.109006 1272 tablet_service.cc:1552] Processing DeleteTablet for tablet 01792fc91bba4ba891812bd25e006268 with delete_type TABLET_DATA_DELETED (replaced by tablet 93927235c4014ddabc37a56b7b265f79) from {username='slave'} at 127.0.0.1:39700
I20260204 02:54:04.124557 2971 tablet_replica.cc:333] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5: stopping tablet replica
I20260204 02:54:04.124828 2971 raft_consensus.cc:2243] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260204 02:54:04.125083 2971 raft_consensus.cc:2272] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260204 02:54:04.125659 2971 ts_tablet_manager.cc:1916] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260204 02:54:04.125651 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260204 02:54:04.138020 2970 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:54:04.138187 2971 ts_tablet_manager.cc:1929] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260204 02:54:04.138306 2971 log.cc:1199] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal/wals/15b2b6a7198c4ebb98daf4de751911a6
I20260204 02:54:04.138213 2970 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:54:04.138659 2971 ts_tablet_manager.cc:1950] T 15b2b6a7198c4ebb98daf4de751911a6 P e2efdafb1246484a98dc5408be6746b5: Deleting consensus metadata
I20260204 02:54:04.139703 2971 tablet_replica.cc:333] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5: stopping tablet replica
I20260204 02:54:04.139887 2971 raft_consensus.cc:2243] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260204 02:54:04.140015 2971 raft_consensus.cc:2272] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260204 02:54:04.140242 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:54:04.140493 2971 ts_tablet_manager.cc:1916] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260204 02:54:04.140674 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:54:04.140779 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:04.141027 2970 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
I20260204 02:54:04.141278 2970 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:54:04.149425 2971 ts_tablet_manager.cc:1929] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260204 02:54:04.149725 2971 log.cc:1199] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5: Deleting WAL directory at /tmp/dist-test-task6hEvLx/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1770173595948734-418-0/minicluster-data/ts-2/wal/wals/01792fc91bba4ba891812bd25e006268
I20260204 02:54:04.150277 2971 ts_tablet_manager.cc:1950] T 01792fc91bba4ba891812bd25e006268 P e2efdafb1246484a98dc5408be6746b5: Deleting consensus metadata
I20260204 02:54:04.250744 1339 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:54:04.252883 2883 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" instance_seqno: 1770173613100203) as {username='slave'} at 127.0.104.131:47939; Asking this server to re-register.
I20260204 02:54:04.253809 1339 heartbeater.cc:461] Registering TS with master...
I20260204 02:54:04.255422 2883 ts_manager.cc:194] Registered new tserver with Master: e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
I20260204 02:54:04.261140 1203 heartbeater.cc:344] Connected to a master server at 127.0.104.188:38789
I20260204 02:54:04.261332 1203 heartbeater.cc:499] Master 127.0.104.188:38789 was elected leader, sending a full tablet report...
I20260204 02:54:04.263054 2883 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" instance_seqno: 1770173612604702) as {username='slave'} at 127.0.104.130:59751; Asking this server to re-register.
I20260204 02:54:04.263731 1203 heartbeater.cc:461] Registering TS with master...
I20260204 02:54:04.264997 2883 ts_manager.cc:194] Registered new tserver with Master: 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:54:04.284672 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 1: this message will repeat every 5th retry.
W20260204 02:54:04.327215 1227 connection.cc:537] server connection from 127.0.0.1:39700 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:54:04.327540 1230 connection.cc:537] client connection to 127.0.104.189:46765 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:54:04.327857 955 connection.cc:537] client connection to 127.0.104.189:46765 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:54:04.328423 2961 negotiation.cc:337] Failed RPC negotiation. Trace:
0204 02:54:04.076490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.0.104.130:44571)
0204 02:54:04.095862 (+ 19372us) server_negotiation.cc:207] Beginning negotiation
0204 02:54:04.095868 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0204 02:54:04.095886 (+ 18us) server_negotiation.cc:408] Connection header received
0204 02:54:04.095967 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0204 02:54:04.095973 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0204 02:54:04.096042 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0204 02:54:04.104733 (+ 8691us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0204 02:54:04.104969 (+ 236us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0204 02:54:04.106523 (+ 1554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0204 02:54:04.327331 (+220808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41484: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":19222,"thread_start_us":19136,"threads_started":1}
W20260204 02:54:04.328500 2964 negotiation.cc:337] Failed RPC negotiation. Trace:
0204 02:54:04.087171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.0.104.129:45767)
0204 02:54:04.087891 (+ 720us) server_negotiation.cc:207] Beginning negotiation
0204 02:54:04.087897 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0204 02:54:04.087916 (+ 19us) server_negotiation.cc:408] Connection header received
0204 02:54:04.087982 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0204 02:54:04.087987 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0204 02:54:04.088069 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0204 02:54:04.097158 (+ 9089us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0204 02:54:04.097456 (+ 298us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0204 02:54:04.098862 (+ 1406us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0204 02:54:04.327753 (+228891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56556: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":504,"threads_started":1}
W20260204 02:54:04.329023 1094 connection.cc:537] client connection to 127.0.104.189:46765 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:54:04.329413 1349 connection.cc:537] client connection to 127.0.104.189:46765 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260204 02:54:04.330191 1349 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.189:46765)
W20260204 02:54:04.330683 1349 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.189:46765)
W20260204 02:54:04.330925 1349 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.189:46765)
W20260204 02:54:04.331120 1349 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.189:46765)
I20260204 02:54:04.580030 2984 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:54:04.580209 2984 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:04.581938 2984 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:54:04.582783 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:54:04.583992 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:54:04.584220 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:54:04.584410 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:54:04.584733 2984 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:54:05.240376 2994 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:54:05.240583 2994 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:05.241317 2994 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:54:05.243027 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:54:05.243289 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:54:05.243438 2871 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:54:05.243733 2994 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:54:06.058049 418 master-stress-test.cc:577] Killing test thread 1/25
I20260204 02:54:06.165205 3008 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:54:06.165416 3008 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:06.165974 3008 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:54:06.168182 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:54:06.168397 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:54:06.168522 2871 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:54:06.168767 3008 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:54:07.295359 3024 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:54:07.295648 3024 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:07.296350 3024 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:54:07.298502 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:54:07.298683 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:54:07.298797 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:54:07.299012 3024 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:54:08.253409 3036 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:54:08.253587 3036 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:08.254184 3036 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:54:08.256104 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:54:08.256492 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:54:08.256589 2871 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:54:08.256790 3036 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:54:08.305497 3031 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:54:08.305697 3031 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:54:08.306463 3031 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:54:08.307255 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:54:08.307494 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:54:08.307667 1091 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:08.307935 3031 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:54:08.951103 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:54:08.951272 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:54:09.847241 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 16: this message will repeat every 5th retry.
W20260204 02:54:10.225585 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 16: this message will repeat every 5th retry.
I20260204 02:54:10.627905 3058 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:54:10.628110 3058 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:54:10.628870 3058 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:54:10.629721 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:54:10.629900 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:54:10.630046 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:10.630332 3058 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:54:11.419404 3064 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:54:11.419631 3064 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:54:11.420296 3064 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:54:11.421126 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:54:11.421285 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:54:11.421435 1091 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:11.421667 3064 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:54:11.727207 3075 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:54:11.727427 3075 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:11.728168 3075 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:54:11.729075 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260204 02:54:11.730381 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:54:11.730557 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:54:11.730686 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:54:11.730904 3075 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260204 02:54:13.655531 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 6: this message will repeat every 5th retry.
I20260204 02:54:13.952813 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:54:13.953049 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:54:14.297940 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 6: this message will repeat every 5th retry.
I20260204 02:54:16.855298 3116 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:54:16.855590 3116 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:16.856397 3116 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:54:16.857484 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:54:16.858772 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:54:16.858959 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:54:16.859107 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:54:16.859411 3116 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:54:18.954337 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:54:18.954528 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:54:19.796470 3144 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:54:19.796689 3144 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:54:19.797552 3144 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:54:19.798444 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:54:19.798647 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:54:19.798791 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:19.799113 3144 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260204 02:54:19.906665 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 21: this message will repeat every 5th retry.
W20260204 02:54:20.297534 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 21: this message will repeat every 5th retry.
I20260204 02:54:21.070694 3148 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:54:21.070891 3148 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:54:21.071705 3148 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:54:21.072463 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:54:21.072669 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:54:21.072753 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:21.072966 3148 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260204 02:54:23.377435 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 11: this message will repeat every 5th retry.
I20260204 02:54:23.956104 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:54:23.956292 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:54:24.173020 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 11: this message will repeat every 5th retry.
I20260204 02:54:24.932878 3200 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:54:24.933075 3200 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:24.933799 3200 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:54:24.934698 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:54:24.935702 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:54:24.935868 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:54:24.936013 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:54:24.936230 3200 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:54:25.651993 3198 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:54:25.652195 3198 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:54:25.652851 3198 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:54:25.653671 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:54:25.653825 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:54:25.653981 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:25.654353 3198 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:54:28.852885 3238 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:54:28.853076 3238 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:28.853650 3238 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:54:28.855834 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:54:28.856115 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:54:28.856261 2871 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:54:28.856531 3238 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:54:28.957522 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:54:28.957772 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:54:29.432649 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 26: this message will repeat every 5th retry.
W20260204 02:54:30.020608 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 26: this message will repeat every 5th retry.
I20260204 02:54:30.411672 3250 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:54:30.411851 3250 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:54:30.412509 3250 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:54:30.413282 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:54:30.413455 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:54:30.413544 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:30.413728 3250 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:54:32.823808 3271 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:54:32.824054 3271 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:54:32.824812 3271 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:54:32.825734 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:54:32.825963 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:54:32.826085 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:32.826305 3271 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260204 02:54:33.141122 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 16: this message will repeat every 5th retry.
W20260204 02:54:33.724130 1205 heartbeater.cc:646] Failed to heartbeat to 127.0.104.189:46765 (89 consecutive failures): Network error: Failed to ping master at 127.0.104.189:46765: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:54:33.724189 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 16: this message will repeat every 5th retry.
W20260204 02:54:33.816320 1340 heartbeater.cc:646] Failed to heartbeat to 127.0.104.190:33671 (112 consecutive failures): Network error: Failed to ping master at 127.0.104.190:33671: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:54:33.959262 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:54:33.959537 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:54:34.056393 1065 heartbeater.cc:646] Failed to heartbeat to 127.0.104.190:33671 (162 consecutive failures): Network error: Failed to ping master at 127.0.104.190:33671: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:54:34.726531 3294 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:54:34.726712 3294 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:54:34.727378 3294 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:54:34.728340 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:54:34.728544 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:54:34.728647 1091 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:34.728960 3294 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:54:38.961277 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:54:38.961474 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:54:39.152102 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 31: this message will repeat every 5th retry.
I20260204 02:54:39.189250 3332 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:54:39.189435 3332 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:54:39.190171 3332 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:54:39.190997 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:54:39.191159 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:54:39.191313 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:39.191588 3332 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:54:40.046975 3344 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:54:40.047163 3344 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:40.047824 3344 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:54:40.048784 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260204 02:54:40.049846 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:54:40.050006 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:54:40.050168 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:54:40.050419 3344 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260204 02:54:40.160876 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 31: this message will repeat every 5th retry.
W20260204 02:54:43.129104 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 21: this message will repeat every 5th retry.
W20260204 02:54:43.505348 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 21: this message will repeat every 5th retry.
I20260204 02:54:43.962675 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:54:43.962900 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:54:46.626757 3398 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:54:46.626940 3398 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:54:46.627871 3398 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:54:46.628700 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:54:46.628872 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:54:46.629017 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:46.629271 3398 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:54:48.964598 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:54:48.964856 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:54:49.381703 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 36: this message will repeat every 5th retry.
W20260204 02:54:50.191660 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 36: this message will repeat every 5th retry.
W20260204 02:54:52.421994 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 26: this message will repeat every 5th retry.
W20260204 02:54:52.774468 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 26: this message will repeat every 5th retry.
I20260204 02:54:53.966449 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:54:53.966675 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:54:55.436600 3494 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:54:55.436842 3494 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:54:55.437748 3494 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:54:55.438757 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:54:55.438995 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:54:55.439148 1091 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:55.439369 3494 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:54:57.866665 3509 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:54:57.866895 3509 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:54:57.867655 3509 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:54:57.868521 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:54:57.868712 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:54:57.868885 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:54:57.869153 3509 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:54:58.968312 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:54:58.968545 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:54:59.366523 3539 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:54:59.366726 3539 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:54:59.367681 3539 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:54:59.368366 2871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:54:59.369549 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:54:59.369730 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:54:59.369844 2871 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:54:59.370054 3539 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260204 02:54:59.459612 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 41: this message will repeat every 5th retry.
W20260204 02:54:59.665365 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 41: this message will repeat every 5th retry.
I20260204 02:55:01.890297 3565 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:55:01.890516 3565 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:55:01.891232 3565 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:55:01.892138 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:55:01.892329 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:55:01.892467 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:55:01.892706 3565 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260204 02:55:02.092132 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 31: this message will repeat every 5th retry.
W20260204 02:55:02.716918 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 31: this message will repeat every 5th retry.
I20260204 02:55:03.969834 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:55:03.970041 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:55:08.971448 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:55:08.971653 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:55:09.005772 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 46: this message will repeat every 5th retry.
W20260204 02:55:10.261931 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 46: this message will repeat every 5th retry.
I20260204 02:55:10.980836 3616 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:55:10.981096 3616 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:55:10.982062 3616 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:55:10.982867 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:55:10.983237 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:55:10.983363 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:55:10.983675 3616 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:55:11.794358 3652 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:55:11.794582 3652 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:55:11.795994 3652 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:55:11.796890 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:55:11.797065 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:55:11.797237 1091 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:55:11.797565 3652 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260204 02:55:12.055784 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 36: this message will repeat every 5th retry.
W20260204 02:55:12.259804 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 36: this message will repeat every 5th retry.
I20260204 02:55:13.976147 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:55:13.976421 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:55:15.287405 3681 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:55:15.287704 3681 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:55:15.288364 3681 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:55:15.289255 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:55:15.289461 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:55:15.289590 1091 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:55:15.289819 3681 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:55:16.520087 3707 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:55:16.520283 3707 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:55:16.521034 3707 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:55:16.522022 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:55:16.523102 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:55:16.523241 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:55:16.523391 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:55:16.523654 3707 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260204 02:55:18.904857 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 51: this message will repeat every 5th retry.
I20260204 02:55:18.977869 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:55:18.978076 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:55:19.953030 3734 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:55:19.953220 3734 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:55:19.953774 3734 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:55:19.954748 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:55:19.954924 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:55:19.955080 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:55:19.955292 3734 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260204 02:55:20.465809 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 51: this message will repeat every 5th retry.
W20260204 02:55:22.338774 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 41: this message will repeat every 5th retry.
W20260204 02:55:22.926154 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 41: this message will repeat every 5th retry.
I20260204 02:55:23.979534 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:55:23.979712 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:55:28.981034 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:55:28.981666 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:55:29.204237 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 56: this message will repeat every 5th retry.
I20260204 02:55:29.478981 3835 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:55:29.479225 3835 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:55:29.480331 3835 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:55:29.481042 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:55:29.481503 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:55:29.481608 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:55:29.481906 3835 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:55:29.846092 3838 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:55:29.846400 3838 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:55:29.847290 3838 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:55:29.848810 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:55:29.850020 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:55:29.850194 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:55:29.850293 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:55:29.850517 3838 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260204 02:55:30.548267 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 56: this message will repeat every 5th retry.
W20260204 02:55:31.712430 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 46: this message will repeat every 5th retry.
W20260204 02:55:32.095606 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 46: this message will repeat every 5th retry.
W20260204 02:55:33.951671 1340 heartbeater.cc:646] Failed to heartbeat to 127.0.104.190:33671 (232 consecutive failures): Network error: Failed to ping master at 127.0.104.190:33671: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:55:33.983057 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:55:33.983281 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:55:34.053548 1204 heartbeater.cc:646] Failed to heartbeat to 127.0.104.190:33671 (343 consecutive failures): Network error: Failed to ping master at 127.0.104.190:33671: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:55:34.114190 1066 heartbeater.cc:646] Failed to heartbeat to 127.0.104.189:46765 (275 consecutive failures): Network error: Failed to ping master at 127.0.104.189:46765: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:55:37.500449 3917 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:55:37.500630 3917 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:55:37.501418 3917 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:55:37.502343 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:55:37.502992 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:55:37.503155 1091 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:55:37.503424 3917 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:55:38.984942 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:55:38.985198 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:55:40.096442 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 61: this message will repeat every 5th retry.
I20260204 02:55:40.437263 3947 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:55:40.437455 3947 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:55:40.438082 3947 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:55:40.439092 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:55:40.439288 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:55:40.439406 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:55:40.439664 3947 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260204 02:55:40.831756 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 61: this message will repeat every 5th retry.
W20260204 02:55:41.796399 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 51: this message will repeat every 5th retry.
W20260204 02:55:42.531523 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 51: this message will repeat every 5th retry.
I20260204 02:55:43.986871 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:55:43.987098 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:55:45.206871 3980 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:55:45.207124 3980 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:55:45.207942 3980 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:55:45.208880 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:55:45.209091 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:55:45.209232 1091 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:55:45.209472 3980 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:55:45.637477 3998 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:55:45.637696 3998 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:55:45.638618 3998 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:55:45.639386 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:55:45.639544 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:55:45.639657 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:55:45.640012 3998 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:55:47.858399 4027 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:55:47.858672 4027 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:55:47.859598 4027 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:55:47.861032 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:55:47.862252 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:55:47.862398 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:55:47.862525 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:55:47.862798 4027 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:55:48.988356 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:55:48.988541 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:55:50.379652 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 66: this message will repeat every 5th retry.
W20260204 02:55:51.361752 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 66: this message will repeat every 5th retry.
W20260204 02:55:51.579160 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 56: this message will repeat every 5th retry.
W20260204 02:55:53.424711 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 56: this message will repeat every 5th retry.
I20260204 02:55:53.989796 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:55:53.989969 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:55:58.991555 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:55:58.991729 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:55:59.389920 4123 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:55:59.390115 4123 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:55:59.390856 4123 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:55:59.391793 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:55:59.391984 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:55:59.392100 1091 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:55:59.392314 4123 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260204 02:55:59.770394 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 71: this message will repeat every 5th retry.
I20260204 02:56:00.631039 4138 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:56:00.631266 4138 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:56:00.632057 4138 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:56:00.633013 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:56:00.633234 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:56:00.633351 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:56:00.633567 4138 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260204 02:56:00.774921 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 61: this message will repeat every 5th retry.
W20260204 02:56:01.155937 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 71: this message will repeat every 5th retry.
W20260204 02:56:03.097708 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 61: this message will repeat every 5th retry.
I20260204 02:56:03.993528 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:56:03.993736 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:56:04.006242 2873 maintenance_manager.cc:419] P 202e4a0debb24c8aa89a4724d72fcf37: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033355
I20260204 02:56:04.070807 2872 maintenance_manager.cc:643] P 202e4a0debb24c8aa89a4724d72fcf37: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.064s user 0.020s sys 0.004s Metrics: {"bytes_written":15235,"cfile_init":1,"dirs.queue_time_us":362,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":696,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":541,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":34,"thread_start_us":485,"threads_started":2,"wal-append.queue_time_us":343}
I20260204 02:56:04.071980 2873 maintenance_manager.cc:419] P 202e4a0debb24c8aa89a4724d72fcf37: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1836 bytes on disk
I20260204 02:56:04.072636 2872 maintenance_manager.cc:643] P 202e4a0debb24c8aa89a4724d72fcf37: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20260204 02:56:04.779873 4194 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:56:04.780138 4194 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:56:04.781023 4194 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:56:04.782035 2871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:56:04.783185 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:56:04.783433 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:56:04.783630 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:56:04.783874 4194 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:56:06.330376 4213 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:56:06.330575 4213 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:56:06.331259 4213 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:56:06.333813 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:56:06.334038 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:56:06.334139 2871 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:56:06.334401 4213 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:56:08.995237 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:56:08.995446 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:56:09.817538 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 76: this message will repeat every 5th retry.
W20260204 02:56:10.662745 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 66: this message will repeat every 5th retry.
W20260204 02:56:11.453325 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 76: this message will repeat every 5th retry.
W20260204 02:56:11.923543 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 66: this message will repeat every 5th retry.
I20260204 02:56:12.292905 4282 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:56:12.293115 4282 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:56:12.293802 4282 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:56:12.294706 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:56:12.294966 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:56:12.295071 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:56:12.295358 4282 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:56:13.996676 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:56:13.996873 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:56:15.273072 4326 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:56:15.273241 4326 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:56:15.274072 4326 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:56:15.274762 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:56:15.274955 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:56:15.275089 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:56:15.275310 4326 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:56:18.998452 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:56:18.998680 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:56:19.953315 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 81: this message will repeat every 5th retry.
W20260204 02:56:19.961133 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 71: this message will repeat every 5th retry.
I20260204 02:56:20.015813 4385 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:56:20.016036 4385 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:56:20.016764 4385 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:56:20.017835 2871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260204 02:56:20.019003 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:56:20.019194 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:56:20.019331 2871 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:56:20.019582 4385 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260204 02:56:21.794126 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 81: this message will repeat every 5th retry.
W20260204 02:56:22.047672 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 71: this message will repeat every 5th retry.
I20260204 02:56:24.000250 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:56:24.000499 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:56:28.978204 4505 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:56:28.978412 4505 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:56:28.979710 4505 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:56:28.980588 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:56:28.981007 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:56:28.981129 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:56:28.981410 4505 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:56:29.002367 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:56:29.002615 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:56:29.886041 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 76: this message will repeat every 5th retry.
W20260204 02:56:30.137348 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 86: this message will repeat every 5th retry.
I20260204 02:56:31.349881 4499 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:56:31.350081 4499 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:56:31.351156 4499 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:56:31.352066 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:56:31.352254 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:56:31.352416 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:56:31.352684 4499 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260204 02:56:31.567618 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 76: this message will repeat every 5th retry.
W20260204 02:56:31.721527 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 86: this message will repeat every 5th retry.
I20260204 02:56:34.004086 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:56:34.004316 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:56:34.093405 1340 heartbeater.cc:646] Failed to heartbeat to 127.0.104.190:33671 (352 consecutive failures): Network error: Failed to ping master at 127.0.104.190:33671: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:56:34.216984 1205 heartbeater.cc:646] Failed to heartbeat to 127.0.104.189:46765 (452 consecutive failures): Network error: Failed to ping master at 127.0.104.189:46765: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:56:34.480134 1066 heartbeater.cc:646] Failed to heartbeat to 127.0.104.189:46765 (463 consecutive failures): Network error: Failed to ping master at 127.0.104.189:46765: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:56:35.815363 4595 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:56:35.815586 4595 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:56:35.816349 4595 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:56:35.817396 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:56:35.817621 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:56:35.817790 1091 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:56:35.818090 4595 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:56:38.128706 4627 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:56:38.128913 4627 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:56:38.129782 4627 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:56:38.130709 2871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:56:38.132136 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:56:38.132301 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:56:38.132459 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:56:38.132704 4627 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:56:39.005889 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:56:39.006135 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:56:39.220039 4638 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:56:39.220245 4638 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:56:39.220937 4638 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:56:39.221879 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:56:39.222059 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:56:39.222203 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:56:39.222434 4638 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260204 02:56:39.462280 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 81: this message will repeat every 5th retry.
W20260204 02:56:39.614811 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 91: this message will repeat every 5th retry.
W20260204 02:56:41.269580 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 91: this message will repeat every 5th retry.
W20260204 02:56:41.905164 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 81: this message will repeat every 5th retry.
I20260204 02:56:44.007802 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:56:44.008026 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:56:47.193003 4751 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:56:47.193243 4751 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:56:47.194031 4751 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:56:47.195652 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:56:47.196866 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:56:47.197031 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:56:47.197177 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:56:47.197470 4751 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:56:49.009800 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:56:49.010010 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:56:49.558454 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 86: this message will repeat every 5th retry.
W20260204 02:56:49.667735 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 96: this message will repeat every 5th retry.
W20260204 02:56:50.539043 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 96: this message will repeat every 5th retry.
W20260204 02:56:52.084134 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 86: this message will repeat every 5th retry.
I20260204 02:56:53.275271 4828 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:56:53.275578 4828 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:56:53.276607 4828 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:56:53.277504 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:56:53.277700 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:56:53.277855 1091 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:56:53.278151 4828 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:56:54.011693 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:56:54.011895 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:56:58.179858 4901 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:56:58.180050 4901 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:56:58.180680 4901 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:56:58.181595 2871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:56:58.182775 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:56:58.182942 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:56:58.183058 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:56:58.183280 4901 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:56:58.224989 4896 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:56:58.225251 4896 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:56:58.226052 4896 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:56:58.227129 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:56:58.227325 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:56:58.227438 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:56:58.227759 4896 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:56:59.013190 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:56:59.013396 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:56:59.366320 4920 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:56:59.366526 4920 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:56:59.367113 4920 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:56:59.368135 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:56:59.368323 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:56:59.368427 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:56:59.368657 4920 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260204 02:56:59.607260 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 101: this message will repeat every 5th retry.
W20260204 02:56:59.694929 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 91: this message will repeat every 5th retry.
I20260204 02:57:00.122893 4929 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:57:00.123083 4929 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:57:00.123782 4929 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:57:00.125697 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:57:00.125900 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:57:00.126001 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:57:00.126255 4929 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260204 02:57:00.238014 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 101: this message will repeat every 5th retry.
W20260204 02:57:01.711886 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 91: this message will repeat every 5th retry.
I20260204 02:57:03.465314 4979 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:57:03.465525 4979 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:57:03.466167 4979 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:57:03.467185 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260204 02:57:03.468394 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:57:03.468525 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:57:03.468688 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:57:03.468974 4979 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:57:04.014882 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:57:04.015129 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:57:08.641238 5050 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:57:08.641499 5050 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:57:08.642455 5050 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:57:08.643544 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:57:08.643776 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:57:08.643920 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:57:08.644194 5050 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:57:08.929281 5044 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:57:08.929527 5044 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:57:08.930200 5044 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:57:08.931109 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:57:08.931262 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:57:08.931380 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:57:08.931654 5044 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:57:09.016669 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:57:09.016932 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:57:10.156006 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 106: this message will repeat every 5th retry.
W20260204 02:57:10.189721 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 96: this message will repeat every 5th retry.
W20260204 02:57:10.677700 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 106: this message will repeat every 5th retry.
W20260204 02:57:11.212216 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 96: this message will repeat every 5th retry.
I20260204 02:57:12.736516 5104 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:57:12.736738 5104 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:57:12.737416 5104 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:57:12.738353 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:57:12.740061 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:57:12.740317 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:57:12.740429 2871 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:57:12.740681 5104 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:57:14.018362 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:57:14.018539 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:57:17.978387 5177 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:57:17.978590 5177 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:57:17.979431 5177 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:57:17.980343 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:57:17.980675 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:57:17.980855 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:57:17.981117 5177 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:57:18.664659 5173 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:57:18.664861 5173 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:57:18.665517 5173 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:57:18.666263 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:57:18.666425 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:57:18.666587 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:57:18.666843 5173 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:57:19.020081 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:57:19.020279 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:57:20.368170 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 101: this message will repeat every 5th retry.
W20260204 02:57:20.505896 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 101: this message will repeat every 5th retry.
W20260204 02:57:20.974148 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 111: this message will repeat every 5th retry.
W20260204 02:57:21.367959 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 111: this message will repeat every 5th retry.
I20260204 02:57:23.125654 5233 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:57:23.125859 5233 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:57:23.126596 5233 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:57:23.127501 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:57:23.128727 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:57:23.128899 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:57:23.128997 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:57:23.129199 5233 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:57:23.201788 5213 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:57:23.202008 5213 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:57:23.202716 5213 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:57:23.203658 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:57:23.203804 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:57:23.203979 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:57:23.204246 5213 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:57:24.021724 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:57:24.021970 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:57:29.023989 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:57:29.024191 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:57:30.376499 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 116: this message will repeat every 5th retry.
I20260204 02:57:30.509509 5326 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:57:30.509727 5326 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:57:30.510509 5326 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:57:30.511405 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:57:30.511603 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:57:30.511747 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:57:30.512012 5326 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260204 02:57:30.584141 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 106: this message will repeat every 5th retry.
W20260204 02:57:30.753486 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 106: this message will repeat every 5th retry.
W20260204 02:57:31.276728 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 116: this message will repeat every 5th retry.
I20260204 02:57:34.001446 5368 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:57:34.001655 5368 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:57:34.002559 5368 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:57:34.003556 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:57:34.003882 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:57:34.004009 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:57:34.004307 5368 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:57:34.025635 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:57:34.025806 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:57:34.241824 1340 heartbeater.cc:646] Failed to heartbeat to 127.0.104.190:33671 (472 consecutive failures): Network error: Failed to ping master at 127.0.104.190:33671: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:57:34.261354 1205 heartbeater.cc:646] Failed to heartbeat to 127.0.104.189:46765 (634 consecutive failures): Network error: Failed to ping master at 127.0.104.189:46765: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:57:34.481587 1066 heartbeater.cc:646] Failed to heartbeat to 127.0.104.189:46765 (649 consecutive failures): Network error: Failed to ping master at 127.0.104.189:46765: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:57:36.189792 5402 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:57:36.190016 5402 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:57:36.190711 5402 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:57:36.191735 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:57:36.193125 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:57:36.193303 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:57:36.193436 2871 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:57:36.193655 5402 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:57:39.026957 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:57:39.027172 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:57:39.939549 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 111: this message will repeat every 5th retry.
W20260204 02:57:40.094705 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 111: this message will repeat every 5th retry.
W20260204 02:57:40.528076 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 121: this message will repeat every 5th retry.
W20260204 02:57:40.612082 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 121: this message will repeat every 5th retry.
I20260204 02:57:41.321135 5452 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:57:41.321305 5452 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:57:41.321940 5452 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:57:41.322700 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:57:41.322927 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:57:41.323050 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:57:41.323314 5452 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:57:44.029580 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:57:44.029808 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:57:45.932476 5517 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:57:45.932683 5517 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:57:45.933315 5517 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:57:45.934267 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:57:45.934445 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:57:45.934577 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:57:45.934876 5517 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:57:48.048839 5536 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:57:48.049010 5536 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:57:48.049813 5536 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:57:48.050637 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:57:48.050793 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:57:48.050922 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:57:48.051182 5536 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:57:49.031363 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:57:49.031592 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:57:50.075577 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 116: this message will repeat every 5th retry.
W20260204 02:57:50.108683 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 116: this message will repeat every 5th retry.
W20260204 02:57:50.531231 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 126: this message will repeat every 5th retry.
W20260204 02:57:51.340778 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 126: this message will repeat every 5th retry.
I20260204 02:57:54.033013 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:57:54.033233 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:57:54.626693 5626 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:57:54.626926 5626 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:57:54.627744 5626 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:57:54.628582 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:57:54.629865 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:57:54.630025 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:57:54.630185 2871 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:57:54.630471 5626 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:57:59.036787 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:57:59.036981 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:57:59.381330 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 121: this message will repeat every 5th retry.
I20260204 02:57:59.386183 5691 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:57:59.386377 5691 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:57:59.386984 5691 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:57:59.389010 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:57:59.389173 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:57:59.389299 2871 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:57:59.389499 5691 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260204 02:58:00.398247 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 121: this message will repeat every 5th retry.
W20260204 02:58:00.592443 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 131: this message will repeat every 5th retry.
W20260204 02:58:01.639348 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 131: this message will repeat every 5th retry.
I20260204 02:58:04.038656 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:58:04.038893 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:58:04.217231 5729 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:58:04.217471 5729 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:58:04.218384 5729 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:58:04.219380 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:58:04.219614 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:58:04.219729 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:58:04.219995 5729 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:58:05.167160 5754 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:58:05.167400 5754 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:58:05.168650 5754 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:58:05.169576 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:58:05.169797 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:58:05.169934 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:58:05.170264 5754 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:58:07.338157 5776 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:58:07.338397 5776 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:58:07.339159 5776 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:58:07.340082 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:58:07.340296 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:58:07.340430 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:58:07.340687 5776 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:58:09.040493 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:58:09.040719 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:58:09.593742 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 126: this message will repeat every 5th retry.
W20260204 02:58:09.957499 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 136: this message will repeat every 5th retry.
I20260204 02:58:10.068006 5814 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:58:10.068264 5814 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:58:10.069012 5814 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:58:10.070289 2871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260204 02:58:10.071605 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:58:10.071853 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:58:10.071954 2871 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:58:10.072228 5814 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:58:11.021680 5827 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:58:11.021942 5827 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:58:11.022642 5827 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:58:11.023561 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:58:11.023729 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:58:11.023854 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:58:11.024166 5827 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260204 02:58:11.071183 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 136: this message will repeat every 5th retry.
W20260204 02:58:11.418629 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 126: this message will repeat every 5th retry.
I20260204 02:58:12.774706 5834 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:58:12.774950 5834 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:58:12.775658 5834 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:58:12.776540 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:58:12.776688 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:58:12.776834 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:58:12.777125 5834 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:58:14.042657 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:58:14.042873 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:58:15.328482 5880 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:58:15.328735 5880 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:58:15.329401 5880 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:58:15.330404 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:58:15.331663 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:58:15.331854 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:58:15.332000 2871 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:58:15.332315 5880 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:58:19.046806 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:58:19.047022 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:58:19.688736 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 131: this message will repeat every 5th retry.
W20260204 02:58:19.726727 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 141: this message will repeat every 5th retry.
I20260204 02:58:21.554445 5956 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:58:21.554669 5956 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:58:21.555438 5956 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:58:21.556409 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:58:21.557556 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:58:21.557726 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:58:21.557860 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:58:21.558082 5956 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260204 02:58:21.875165 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 141: this message will repeat every 5th retry.
W20260204 02:58:21.875427 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 131: this message will repeat every 5th retry.
I20260204 02:58:23.261916 5974 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:58:23.262127 5974 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:58:23.262859 5974 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:58:23.263779 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:58:23.263955 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:58:23.264088 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:58:23.264313 5974 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:58:23.661406 5981 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:58:23.661598 5981 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:58:23.662346 5981 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:58:23.663069 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:58:23.663213 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:58:23.663339 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:58:23.663578 5981 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:58:24.048436 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:58:24.048647 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:58:26.575300 5997 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:58:26.575610 5997 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:58:26.576287 5997 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:58:26.577183 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:58:26.577355 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:58:26.577508 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:58:26.577765 5997 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260204 02:58:28.615049 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 146: this message will repeat every 5th retry.
W20260204 02:58:28.956578 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 136: this message will repeat every 5th retry.
I20260204 02:58:29.050285 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:58:29.050508 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:58:31.773463 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 136: this message will repeat every 5th retry.
W20260204 02:58:32.467003 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 146: this message will repeat every 5th retry.
I20260204 02:58:33.003111 6092 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:58:33.003304 6092 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:58:33.004099 6092 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:58:33.005052 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:58:33.005203 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:58:33.005342 952 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:58:33.005609 6092 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260204 02:58:34.052016 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:58:34.052241 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:58:34.385066 1340 heartbeater.cc:646] Failed to heartbeat to 127.0.104.190:33671 (592 consecutive failures): Network error: Failed to ping master at 127.0.104.190:33671: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:58:34.456182 1204 heartbeater.cc:646] Failed to heartbeat to 127.0.104.190:33671 (885 consecutive failures): Network error: Failed to ping master at 127.0.104.190:33671: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:58:34.619881 1065 heartbeater.cc:646] Failed to heartbeat to 127.0.104.190:33671 (900 consecutive failures): Network error: Failed to ping master at 127.0.104.190:33671: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:58:35.258108 6129 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:58:35.258337 6129 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:58:35.259133 6129 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:58:35.260190 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:58:35.261410 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:58:35.261584 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:58:35.261685 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:58:35.261924 6129 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260204 02:58:38.722010 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 141: this message will repeat every 5th retry.
W20260204 02:58:38.811219 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 151: this message will repeat every 5th retry.
I20260204 02:58:39.053692 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:58:39.053933 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:58:41.681826 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 141: this message will repeat every 5th retry.
I20260204 02:58:41.893013 6214 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:58:41.893193 6214 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:58:41.894011 6214 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:58:41.894965 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:58:41.895082 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:58:41.895215 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:58:41.895515 6214 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260204 02:58:42.125211 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 151: this message will repeat every 5th retry.
I20260204 02:58:44.055447 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:58:44.055634 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:58:46.495405 6268 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:58:46.495683 6268 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:58:46.496416 6268 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:58:46.497380 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:58:46.497491 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:58:46.497622 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:58:46.497839 6268 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260204 02:58:48.076272 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 156: this message will repeat every 5th retry.
I20260204 02:58:48.635317 6306 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:58:48.635591 6306 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:58:48.636298 6306 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:58:48.637230 2871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:58:48.638424 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:58:48.638643 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:58:48.638733 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:58:48.638958 6306 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260204 02:58:48.788376 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 146: this message will repeat every 5th retry.
I20260204 02:58:49.056792 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:58:49.056990 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260204 02:58:52.005770 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 146: this message will repeat every 5th retry.
W20260204 02:58:52.103775 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 156: this message will repeat every 5th retry.
I20260204 02:58:53.380012 6368 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:58:53.380214 6368 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:58:53.380932 6368 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:58:53.381666 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:58:53.381817 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:58:53.381942 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:58:53.382184 6357 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260204 02:58:54.058490 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:58:54.058706 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:58:56.878193 6407 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
I20260204 02:58:56.878435 6407 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:58:56.879035 6407 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
W20260204 02:58:56.879853 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:58:56.879985 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:58:56.880133 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:58:56.880395 6407 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260204 02:58:57.315900 1093 consensus_peers.cc:597] T 6f0852d9159a45a69935c76182584be3 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0852d9159a45a69935c76182584be3. This is attempt 161: this message will repeat every 5th retry.
I20260204 02:58:58.417088 6435 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:58:58.417302 6435 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:58:58.418064 6435 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:58:58.418962 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:58:58.420127 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:58:58.420275 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
I20260204 02:58:58.420451 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:58:58.420720 6435 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260204 02:58:58.820204 1093 consensus_peers.cc:597] T 01792fc91bba4ba891812bd25e006268 P 4dc2992e324049698330803c9ed2e3f8 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01792fc91bba4ba891812bd25e006268. This is attempt 151: this message will repeat every 5th retry.
I20260204 02:58:59.060091 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:58:59.060340 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:58:59.620971 6441 raft_consensus.cc:493] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4dc2992e324049698330803c9ed2e3f8)
I20260204 02:58:59.621191 6441 raft_consensus.cc:515] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } }
I20260204 02:58:59.621913 6441 leader_election.cc:290] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247), 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571)
W20260204 02:58:59.622838 952 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 4dc2992e324049698330803c9ed2e3f8 (127.0.104.130:44571): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
W20260204 02:58:59.622987 954 leader_election.cc:343] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 8c77d21f66a640d88bf1865d6176c969
I20260204 02:58:59.623114 954 leader_election.cc:304] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3; no voters: 4dc2992e324049698330803c9ed2e3f8, e2efdafb1246484a98dc5408be6746b5
I20260204 02:58:59.623348 6441 raft_consensus.cc:2749] T 8c77d21f66a640d88bf1865d6176c969 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260204 02:59:01.931741 954 consensus_peers.cc:597] T f66e2750d8c343c0bfa567fd36bc4da0 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66e2750d8c343c0bfa567fd36bc4da0. This is attempt 161: this message will repeat every 5th retry.
W20260204 02:59:01.935278 954 consensus_peers.cc:597] T 15b2b6a7198c4ebb98daf4de751911a6 P 8d7605b7e0f949b3ba7ffaddd1ca8db3 -> Peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Couldn't send request to peer e2efdafb1246484a98dc5408be6746b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15b2b6a7198c4ebb98daf4de751911a6. This is attempt 151: this message will repeat every 5th retry.
I20260204 02:59:03.857781 6504 raft_consensus.cc:493] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 78eb37be156448e8879b0df55832ab2d)
I20260204 02:59:03.858100 6504 raft_consensus.cc:515] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e85e1dc6a5d4568ad365ba1d77398ff" member_type: VOTER last_known_addr { host: "127.0.104.190" port: 33671 } } peers { permanent_uuid: "78eb37be156448e8879b0df55832ab2d" member_type: VOTER last_known_addr { host: "127.0.104.189" port: 46765 } } peers { permanent_uuid: "202e4a0debb24c8aa89a4724d72fcf37" member_type: VOTER last_known_addr { host: "127.0.104.188" port: 38789 } }
I20260204 02:59:03.858835 6504 leader_election.cc:290] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671), 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765)
W20260204 02:59:03.860833 2870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260204 02:59:03.862305 2870 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3e85e1dc6a5d4568ad365ba1d77398ff (127.0.104.190:33671): Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111)
W20260204 02:59:03.862476 2871 leader_election.cc:336] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 78eb37be156448e8879b0df55832ab2d (127.0.104.189:46765): Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
I20260204 02:59:03.862643 2871 leader_election.cc:304] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202e4a0debb24c8aa89a4724d72fcf37; no voters: 3e85e1dc6a5d4568ad365ba1d77398ff, 78eb37be156448e8879b0df55832ab2d
I20260204 02:59:03.862910 6504 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 202e4a0debb24c8aa89a4724d72fcf37 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260204 02:59:04.061478 2847 master_runner.cc:186] Error getting master registration for 127.0.104.190:33671: Network error: Client connection negotiation failed: client connection to 127.0.104.190:33671: connect: Connection refused (error 111),
I20260204 02:59:04.061831 2847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260204 02:59:04.103116 2873 maintenance_manager.cc:419] P 202e4a0debb24c8aa89a4724d72fcf37: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1836 bytes on disk
I20260204 02:59:04.108680 2872 maintenance_manager.cc:643] P 202e4a0debb24c8aa89a4724d72fcf37: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.005s sys 0.000s Metrics: {"dirs.queue_time_us":471,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":123,"thread_start_us":607,"threads_started":2,"wal-append.queue_time_us":377}
W20260204 02:59:04.552356 1348 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:59:04.552682 1348 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:59:04.552826 1348 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
W20260204 02:59:04.552949 1348 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
F20260204 02:59:04.563822 1380 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Aborted at 1770173944 (unix time) try "date -d @1770173944" if you are using GNU date ***
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569200 1355 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e971e55de1b44441b34f0913c0d665a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569200 1355 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e971e55de1b44441b34f0913c0d665a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569211 1365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569200 1355 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e971e55de1b44441b34f0913c0d665a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569211 1365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569332 1377 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569200 1355 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e971e55de1b44441b34f0913c0d665a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569211 1365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569332 1377 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570163 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f77bbc9c07244dcb50a3b82a1fc8f1a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569200 1355 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e971e55de1b44441b34f0913c0d665a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569211 1365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569332 1377 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570163 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f77bbc9c07244dcb50a3b82a1fc8f1a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570286 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e7eb44e87e2543d69bd03454b10c8310 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000001a2) received by PID 418 (TID 0x7f189902b700) from PID 418; stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569200 1355 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e971e55de1b44441b34f0913c0d665a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569211 1365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569332 1377 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570163 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f77bbc9c07244dcb50a3b82a1fc8f1a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570286 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e7eb44e87e2543d69bd03454b10c8310 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571452 1371 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569200 1355 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e971e55de1b44441b34f0913c0d665a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569211 1365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569332 1377 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570163 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f77bbc9c07244dcb50a3b82a1fc8f1a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570286 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e7eb44e87e2543d69bd03454b10c8310 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571452 1371 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571560 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b896155e926a4679a475f25f74e33530 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569200 1355 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e971e55de1b44441b34f0913c0d665a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569211 1365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569332 1377 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570163 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f77bbc9c07244dcb50a3b82a1fc8f1a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570286 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e7eb44e87e2543d69bd03454b10c8310 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571452 1371 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571560 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b896155e926a4679a475f25f74e33530 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571769 1363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_237c40ddb38c410c9ce533b72060712f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569200 1355 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e971e55de1b44441b34f0913c0d665a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569211 1365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569332 1377 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570163 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f77bbc9c07244dcb50a3b82a1fc8f1a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570286 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e7eb44e87e2543d69bd03454b10c8310 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571452 1371 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571560 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b896155e926a4679a475f25f74e33530 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571769 1363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_237c40ddb38c410c9ce533b72060712f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571777 1381 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571836 1372 master-stress-test.cc:351] Check failed: _s.ok() Bad status: T*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569200 1355 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e971e55de1b44441b34f0913c0d665a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569211 1365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569332 1377 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570163 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f77bbc9c07244dcb50a3b82a1fc8f1a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570286 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e7eb44e87e2543d69bd03454b10c8310 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571452 1371 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571560 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b896155e926a4679a475f25f74e33530 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571769 1363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_237c40ddb38c410c9ce533b72060712f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571777 1381 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571836 1372 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569200 1355 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e971e55de1b44441b34f0913c0d665a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569211 1365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569332 1377 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570163 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f77bbc9c07244dcb50a3b82a1fc8f1a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570286 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e7eb44e87e2543d69bd03454b10c8310 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571452 1371 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571560 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b896155e926a4679a475f25f74e33530 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571769 1363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_237c40ddb38c410c9ce533b72060712f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571777 1381 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571836 1372 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.572008 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c26f0462e2664a018e18bf0ef613c017 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569200 1355 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e971e55de1b44441b34f0913c0d665a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569211 1365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569332 1377 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570163 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f77bbc9c07244dcb50a3b82a1fc8f1a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570286 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e7eb44e87e2543d69bd03454b10c8310 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571452 1371 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571560 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b896155e926a4679a475f25f74e33530 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571769 1363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_237c40ddb38c410c9ce533b72060712f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571777 1381 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571836 1372 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.572008 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c26f0462e2664a018e18bf0ef613c017 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.572126 1368 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f18b7cc7980 (unknown) at ??:0
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569200 1355 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e971e55de1b44441b34f0913c0d665a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569211 1365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569332 1377 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570163 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f77bbc9c07244dcb50a3b82a1fc8f1a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570286 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e7eb44e87e2543d69bd03454b10c8310 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571452 1371 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571560 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b896155e926a4679a475f25f74e33530 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571769 1363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_237c40ddb38c410c9ce533b72060712f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571777 1381 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571836 1372 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.572008 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c26f0462e2664a018e18bf0ef613c017 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.572126 1368 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.573320 1375 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260204 02:59:04.563947 1366 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.563971 1354 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a2c8b05d2d724f50aaf92bddb8a266e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565184 1376 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565304 1373 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565460 1369 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565531 1370 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.565639 1360 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc4536e3b1e64cb7bb0d82dfb42df2db on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.567691 1367 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569136 1357 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6bba790402724856976fb9e10e3c48fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569200 1355 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e971e55de1b44441b34f0913c0d665a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569211 1365 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.569332 1377 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570163 1359 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3f77bbc9c07244dcb50a3b82a1fc8f1a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.570286 1362 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e7eb44e87e2543d69bd03454b10c8310 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571452 1371 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571560 1361 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b896155e926a4679a475f25f74e33530 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571769 1363 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_237c40ddb38c410c9ce533b72060712f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571777 1381 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.571836 1372 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.572008 1356 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c26f0462e2664a018e18bf0ef613c017 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.572126 1368 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.573320 1375 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)F20260204 02:59:04.573519 1364 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdc0d01a86b043beb3441b2078794538 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.104.190:33671,127.0.104.189:46765,127.0.104.188:38789, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.104.189:46765: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f18b0a9afb7 gsignal at ??:0
@ 0x7f18b0a9c921 abort at ??:0
@ 0x7f18b37d2dcd google::LogMessage::Fail() at ??:0
@ 0x7f18b37d6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f18b37d27cc google::LogMessage::Flush() at ??:0
@ 0x7f18b37d3f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40d9f5 kudu::MasterStressTest::ReplaceTabletThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:391
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
I20260204 02:59:04.583194 6492 raft_consensus.cc:493] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d7605b7e0f949b3ba7ffaddd1ca8db3)
@ 0x7f18b11226df (unknown) at ??:0
I20260204 02:59:04.583393 6492 raft_consensus.cc:515] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7605b7e0f949b3ba7ffaddd1ca8db3" member_type: VOTER last_known_addr { host: "127.0.104.129" port: 45767 } } peers { permanent_uuid: "4dc2992e324049698330803c9ed2e3f8" member_type: VOTER last_known_addr { host: "127.0.104.130" port: 44571 } } peers { permanent_uuid: "e2efdafb1246484a98dc5408be6746b5" member_type: VOTER last_known_addr { host: "127.0.104.131" port: 34247 } }
I20260204 02:59:04.584478 6492 leader_election.cc:290] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767), e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247)
@ 0x7f18b7cbc6db start_thread at ??:0
W20260204 02:59:04.585556 1091 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7605b7e0f949b3ba7ffaddd1ca8db3 (127.0.104.129:45767): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
W20260204 02:59:04.585692 1093 leader_election.cc:343] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2efdafb1246484a98dc5408be6746b5 (127.0.104.131:34247): Not found: Tablet not found: 9a80cbfc27b54a5d945683c39020a027
I20260204 02:59:04.585795 1093 leader_election.cc:304] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dc2992e324049698330803c9ed2e3f8; no voters: 8d7605b7e0f949b3ba7ffaddd1ca8db3, e2efdafb1246484a98dc5408be6746b5
I20260204 02:59:04.586082 6492 raft_consensus.cc:2749] T 9a80cbfc27b54a5d945683c39020a027 P 4dc2992e324049698330803c9ed2e3f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
@ 0x7f18b0b7d71f clone at ??:0