Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251029 16:33:41.761132 6865 test_util.cc:276] Using random seed: -1308352450
2025-10-29T16:33:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-29T16:33:41Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskoTseiL/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskoTseiL/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/security.properties
2025-10-29 16:33:48: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskoTseiL/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskoTseiL/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20251029 16:33:56.025113 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.126:36819
--webserver_interface=127.6.180.126
--webserver_port=0
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251029 16:33:56.379087 7108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:33:56.379444 7108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:33:56.379521 7108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:33:56.387928 7108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 16:33:56.388072 7108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:33:56.388130 7108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 16:33:56.388170 7108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 16:33:56.398619 7108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.126:36819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.180.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:33:56.400449 7108 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:33:56.402405 7108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 16:33:56.411763 7113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:33:56.411789 7114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:33:56.412762 7116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:33:56.413123 7108 server_base.cc:1047] running on GCE node
I20251029 16:33:56.414001 7108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 16:33:56.415773 7108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:33:56.416997 7108 hybrid_clock.cc:648] HybridClock initialized: now 1761755636416967 us; error 63 us; skew 500 ppm
I20251029 16:33:56.417430 7108 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:33:56.419868 7108 webserver.cc:492] Webserver started at http://127.6.180.126:40759/ using document root <none> and password file <none>
I20251029 16:33:56.420758 7108 fs_manager.cc:362] Metadata directory not provided
I20251029 16:33:56.420871 7108 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:33:56.421268 7108 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 16:33:56.425400 7108 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data/instance:
uuid: "0442da13a653435dba87fc90d8e30d1f"
format_stamp: "Formatted at 2025-10-29 16:33:56 on dist-test-slave-l1bh"
I20251029 16:33:56.426282 7108 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/wal/instance:
uuid: "0442da13a653435dba87fc90d8e30d1f"
format_stamp: "Formatted at 2025-10-29 16:33:56 on dist-test-slave-l1bh"
I20251029 16:33:56.432528 7108 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20251029 16:33:56.436625 7122 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:33:56.438225 7108 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251029 16:33:56.438455 7108 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/wal
uuid: "0442da13a653435dba87fc90d8e30d1f"
format_stamp: "Formatted at 2025-10-29 16:33:56 on dist-test-slave-l1bh"
I20251029 16:33:56.438699 7108 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:33:56.460420 7108 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 16:33:56.461441 7108 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:33:56.461755 7108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 16:33:56.483538 7108 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.126:36819
I20251029 16:33:56.483559 7173 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.126:36819 every 8 connection(s)
I20251029 16:33:56.485386 7108 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data/info.pb
I20251029 16:33:56.486685 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 7108
I20251029 16:33:56.487033 6865 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/wal/instance
I20251029 16:33:56.490718 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.125:44441
--webserver_interface=127.6.180.125
--webserver_port=0
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251029 16:33:56.491849 7174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:33:56.499648 7174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } has no permanent_uuid. Determining permanent_uuid...
I20251029 16:33:56.518764 7174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } has no permanent_uuid. Determining permanent_uuid...
W20251029 16:33:56.520185 7126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
W20251029 16:33:56.522032 7174 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.180.125:44441: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:33:56.571501 7174 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } attempt: 1
W20251029 16:33:56.573601 7174 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.180.125:44441: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:33:56.641880 7174 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } attempt: 2
W20251029 16:33:56.644032 7174 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.180.125:44441: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:33:56.735327 7174 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } attempt: 3
W20251029 16:33:56.737743 7174 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.180.125:44441: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
W20251029 16:33:56.848435 7175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:33:56.848798 7175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:33:56.848929 7175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:33:56.857329 7175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 16:33:56.857460 7175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:33:56.857517 7175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 16:33:56.857556 7175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 16:33:56.867630 7175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.125:44441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.180.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:33:56.869427 7175 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:33:56.871553 7175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 16:33:56.880496 7183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:33:56.880527 7184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:33:56.880980 7174 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } attempt: 4
W20251029 16:33:56.882189 7186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:33:56.882297 7175 server_base.cc:1047] running on GCE node
I20251029 16:33:56.883292 7175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20251029 16:33:56.883309 7174 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.180.125:44441: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:33:56.884979 7175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:33:56.886214 7175 hybrid_clock.cc:648] HybridClock initialized: now 1761755636886153 us; error 50 us; skew 500 ppm
I20251029 16:33:56.886602 7175 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:33:56.889030 7175 webserver.cc:492] Webserver started at http://127.6.180.125:41231/ using document root <none> and password file <none>
I20251029 16:33:56.889864 7175 fs_manager.cc:362] Metadata directory not provided
I20251029 16:33:56.889971 7175 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:33:56.890365 7175 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 16:33:56.892958 7175 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/instance:
uuid: "3155690d4bb54c97b5a1e7881608ae08"
format_stamp: "Formatted at 2025-10-29 16:33:56 on dist-test-slave-l1bh"
I20251029 16:33:56.893826 7175 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal/instance:
uuid: "3155690d4bb54c97b5a1e7881608ae08"
format_stamp: "Formatted at 2025-10-29 16:33:56 on dist-test-slave-l1bh"
I20251029 16:33:56.900022 7175 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20251029 16:33:56.904028 7192 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:33:56.905616 7175 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251029 16:33:56.905826 7175 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
uuid: "3155690d4bb54c97b5a1e7881608ae08"
format_stamp: "Formatted at 2025-10-29 16:33:56 on dist-test-slave-l1bh"
I20251029 16:33:56.906059 7175 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:33:56.931653 7175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 16:33:56.932641 7175 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:33:56.932926 7175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 16:33:56.953812 7175 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.125:44441
I20251029 16:33:56.953831 7243 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.125:44441 every 8 connection(s)
I20251029 16:33:56.955500 7175 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
I20251029 16:33:56.960032 7244 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:33:56.962103 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 7175
I20251029 16:33:56.962335 6865 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal/instance
I20251029 16:33:56.963824 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.124:39271
--webserver_interface=127.6.180.124
--webserver_port=0
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251029 16:33:56.969340 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } has no permanent_uuid. Determining permanent_uuid...
I20251029 16:33:56.984113 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } has no permanent_uuid. Determining permanent_uuid...
I20251029 16:33:56.994117 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } has no permanent_uuid. Determining permanent_uuid...
W20251029 16:33:56.995251 7194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
W20251029 16:33:56.996925 7244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.180.124:39271: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:33:57.046332 7244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } attempt: 1
W20251029 16:33:57.048476 7244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.180.124:39271: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:33:57.116842 7244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } attempt: 2
W20251029 16:33:57.119217 7244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.180.124:39271: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:33:57.182623 7174 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } attempt: 5
I20251029 16:33:57.189327 7174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } has no permanent_uuid. Determining permanent_uuid...
W20251029 16:33:57.191200 7174 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.180.124:39271: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:33:57.210489 7244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } attempt: 3
W20251029 16:33:57.212646 7244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.180.124:39271: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:33:57.242507 7174 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } attempt: 1
W20251029 16:33:57.244827 7174 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.180.124:39271: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:33:57.313153 7174 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } attempt: 2
W20251029 16:33:57.315521 7174 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.180.124:39271: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
W20251029 16:33:57.322746 7245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:33:57.323102 7245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:33:57.323172 7245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:33:57.331434 7245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 16:33:57.331550 7245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:33:57.331621 7245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 16:33:57.331660 7245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 16:33:57.341894 7245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.124:39271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.180.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:33:57.343750 7245 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:33:57.345847 7245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251029 16:33:57.355870 7244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } attempt: 4
W20251029 16:33:57.357478 7253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:33:57.357954 7254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:33:57.358340 7244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.180.124:39271: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
W20251029 16:33:57.358917 7256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:33:57.359414 7245 server_base.cc:1047] running on GCE node
I20251029 16:33:57.360388 7245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 16:33:57.362022 7245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:33:57.363251 7245 hybrid_clock.cc:648] HybridClock initialized: now 1761755637363200 us; error 47 us; skew 500 ppm
I20251029 16:33:57.363638 7245 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:33:57.366119 7245 webserver.cc:492] Webserver started at http://127.6.180.124:46685/ using document root <none> and password file <none>
I20251029 16:33:57.366958 7245 fs_manager.cc:362] Metadata directory not provided
I20251029 16:33:57.367082 7245 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:33:57.367486 7245 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 16:33:57.370100 7245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/instance:
uuid: "29f9bab9976745478595681935778cac"
format_stamp: "Formatted at 2025-10-29 16:33:57 on dist-test-slave-l1bh"
I20251029 16:33:57.370977 7245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal/instance:
uuid: "29f9bab9976745478595681935778cac"
format_stamp: "Formatted at 2025-10-29 16:33:57 on dist-test-slave-l1bh"
I20251029 16:33:57.377177 7245 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20251029 16:33:57.380913 7262 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:33:57.382403 7245 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251029 16:33:57.382609 7245 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
uuid: "29f9bab9976745478595681935778cac"
format_stamp: "Formatted at 2025-10-29 16:33:57 on dist-test-slave-l1bh"
I20251029 16:33:57.382833 7245 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:33:57.396116 7245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 16:33:57.397068 7245 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:33:57.397358 7245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 16:33:57.418365 7245 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.124:39271
I20251029 16:33:57.418386 7314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.124:39271 every 8 connection(s)
I20251029 16:33:57.420245 7245 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
I20251029 16:33:57.421753 7174 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } attempt: 3
I20251029 16:33:57.425678 7315 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:33:57.426595 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 7245
I20251029 16:33:57.426826 6865 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal/instance
I20251029 16:33:57.435190 7315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } has no permanent_uuid. Determining permanent_uuid...
I20251029 16:33:57.446955 7174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: Bootstrap starting.
I20251029 16:33:57.448438 7315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } has no permanent_uuid. Determining permanent_uuid...
I20251029 16:33:57.452096 7174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: Neither blocks nor log segments found. Creating new log.
I20251029 16:33:57.454794 7174 log.cc:826] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: Log is configured to *not* fsync() on all Append() calls
I20251029 16:33:57.457481 7315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } has no permanent_uuid. Determining permanent_uuid...
I20251029 16:33:57.458482 7174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: No bootstrap required, opened a new log
I20251029 16:33:57.466145 7174 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:57.466601 7174 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:33:57.466728 7174 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0442da13a653435dba87fc90d8e30d1f, State: Initialized, Role: FOLLOWER
I20251029 16:33:57.467464 7174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:57.468575 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:57.468932 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:33:57.469751 7174 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: configured and running, proceeding with master startup.
I20251029 16:33:57.470321 7315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Bootstrap starting.
I20251029 16:33:57.474407 7315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Neither blocks nor log segments found. Creating new log.
I20251029 16:33:57.475600 7315 log.cc:826] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Log is configured to *not* fsync() on all Append() calls
I20251029 16:33:57.478511 7315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: No bootstrap required, opened a new log
I20251029 16:33:57.486634 7315 raft_consensus.cc:359] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:57.487118 7315 raft_consensus.cc:385] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:33:57.487283 7315 raft_consensus.cc:740] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f9bab9976745478595681935778cac, State: Initialized, Role: FOLLOWER
I20251029 16:33:57.488235 7315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:57.494187 7315 sys_catalog.cc:565] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: configured and running, proceeding with master startup.
I20251029 16:33:57.492632 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:57.499492 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:33:57.520610 7342 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251029 16:33:57.526013 7108 master_runner.cc:186] Error getting master registration for 127.6.180.125:44441: OK, instance_id { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" instance_seqno: 1761755636948334 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251029 16:33:57.535135 7108 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:33:57.549737 7356 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251029 16:33:57.557508 7245 master_runner.cc:186] Error getting master registration for 127.6.180.125:44441: OK, instance_id { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" instance_seqno: 1761755636948334 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251029 16:33:57.564854 7245 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:33:57.641110 7340 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:33:57.641417 7340 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:57.642871 7340 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
I20251029 16:33:57.643893 7149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0442da13a653435dba87fc90d8e30d1f" is_pre_election: true
I20251029 16:33:57.644518 7149 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f9bab9976745478595681935778cac in term 0.
I20251029 16:33:57.645165 7264 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac; no voters:
I20251029 16:33:57.645584 7340 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 16:33:57.645766 7340 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 16:33:57.645871 7340 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:33:57.649163 7340 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
W20251029 16:33:57.649610 7268 tablet.cc:2378] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 16:33:57.650619 7149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0442da13a653435dba87fc90d8e30d1f"
I20251029 16:33:57.650866 7149 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:33:57.651465 7340 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 1 election: Requested vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
I20251029 16:33:57.654418 7149 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f9bab9976745478595681935778cac in term 1.
I20251029 16:33:57.654976 7264 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac; no voters:
I20251029 16:33:57.655385 7340 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 1 FOLLOWER]: Leader election won for term 1
I20251029 16:33:57.655942 7340 raft_consensus.cc:697] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 1 LEADER]: Becoming Leader. State: Replica: 29f9bab9976745478595681935778cac, State: Running, Role: LEADER
I20251029 16:33:57.656538 7340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:57.657534 7244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } attempt: 5
I20251029 16:33:57.660387 7357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f9bab9976745478595681935778cac. Latest consensus state: current_term: 1 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:57.660773 7357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: LEADER
I20251029 16:33:57.661844 7359 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251029 16:33:57.667857 7359 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251029 16:33:57.674667 7244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Bootstrap starting.
I20251029 16:33:57.680517 7244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Neither blocks nor log segments found. Creating new log.
I20251029 16:33:57.682111 7244 log.cc:826] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Log is configured to *not* fsync() on all Append() calls
I20251029 16:33:57.683848 7149 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 1 FOLLOWER]: Refusing update from remote peer 29f9bab9976745478595681935778cac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 16:33:57.685060 7340 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:33:57.686662 7244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: No bootstrap required, opened a new log
I20251029 16:33:57.699481 7244 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:57.702133 7244 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:33:57.702445 7244 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3155690d4bb54c97b5a1e7881608ae08, State: Initialized, Role: FOLLOWER
I20251029 16:33:57.703697 7244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:57.705636 7365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:57.706292 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f9bab9976745478595681935778cac. Latest consensus state: current_term: 1 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:57.706379 7365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:33:57.706532 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:33:57.706918 7244 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: configured and running, proceeding with master startup.
I20251029 16:33:57.707348 7217 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:33:57.711757 7217 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 1 FOLLOWER]: Refusing update from remote peer 29f9bab9976745478595681935778cac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 16:33:57.712065 7361 mvcc.cc:204] Tried to move back new op lower bound from 7216151091932364800 to 7216151091846369280. Current Snapshot: MvccSnapshot[applied={T|T < 7216151091932364800}]
I20251029 16:33:57.712630 7218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3155690d4bb54c97b5a1e7881608ae08"
I20251029 16:33:57.712845 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:57.713089 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:33:57.713689 7219 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3155690d4bb54c97b5a1e7881608ae08" is_pre_election: true
I20251029 16:33:57.715265 7340 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:33:57.724283 7359 catalog_manager.cc:1357] Generated new cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:33:57.724478 7359 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251029 16:33:57.724479 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:57.724910 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: LEADER
I20251029 16:33:57.751435 7374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:57.751507 7365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f9bab9976745478595681935778cac. Latest consensus state: current_term: 1 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:57.751937 7374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:33:57.751969 7365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:33:57.766301 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:57.766770 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: LEADER
I20251029 16:33:57.775260 7359 catalog_manager.cc:1380] Generated new certificate authority record
I20251029 16:33:57.777158 7359 catalog_manager.cc:1514] Loading token signing keys...
I20251029 16:33:57.804517 7359 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Generated new TSK 0
I20251029 16:33:57.805896 7359 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251029 16:33:57.808486 7388 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:33:57.808723 7388 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: loading cluster ID for follower catalog manager: success
I20251029 16:33:57.809787 7359 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251029 16:33:57.811425 7359 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251029 16:33:57.813050 7388 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: acquiring CA information for follower catalog manager: success
I20251029 16:33:57.816104 7388 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 16:33:57.841903 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.180.65:0
--local_ip_for_outbound_sockets=127.6.180.65
--webserver_interface=127.6.180.65
--webserver_port=0
--tserver_master_addrs=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251029 16:33:58.303436 7391 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29f9bab9976745478595681935778cac)
I20251029 16:33:58.303750 7391 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:58.305226 7391 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
I20251029 16:33:58.306031 7148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3155690d4bb54c97b5a1e7881608ae08" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0442da13a653435dba87fc90d8e30d1f" is_pre_election: true
I20251029 16:33:58.306128 7289 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3155690d4bb54c97b5a1e7881608ae08" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "29f9bab9976745478595681935778cac" is_pre_election: true
I20251029 16:33:58.306331 7148 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3155690d4bb54c97b5a1e7881608ae08 in term 1.
I20251029 16:33:58.307375 7194 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08; no voters: 29f9bab9976745478595681935778cac
I20251029 16:33:58.307798 7391 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251029 16:33:58.308015 7391 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 29f9bab9976745478595681935778cac)
I20251029 16:33:58.308197 7391 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 1 FOLLOWER]: Advancing to term 2
I20251029 16:33:58.310613 7391 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:58.311450 7391 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 2 election: Requested vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
I20251029 16:33:58.312287 7289 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3155690d4bb54c97b5a1e7881608ae08" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "29f9bab9976745478595681935778cac"
I20251029 16:33:58.312371 7148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3155690d4bb54c97b5a1e7881608ae08" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0442da13a653435dba87fc90d8e30d1f"
I20251029 16:33:58.312569 7148 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 1 FOLLOWER]: Advancing to term 2
I20251029 16:33:58.314780 7148 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3155690d4bb54c97b5a1e7881608ae08 in term 2.
I20251029 16:33:58.315521 7194 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08; no voters: 29f9bab9976745478595681935778cac
I20251029 16:33:58.316370 7391 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 2 FOLLOWER]: Leader election won for term 2
I20251029 16:33:58.329468 7391 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 2 LEADER]: Becoming Leader. State: Replica: 3155690d4bb54c97b5a1e7881608ae08, State: Running, Role: LEADER
I20251029 16:33:58.330081 7391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:58.331677 7217 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 2 LEADER]: Rejecting Update request from peer 29f9bab9976745478595681935778cac for earlier term 1. Current term is 2. Ops: []
I20251029 16:33:58.332508 7393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3155690d4bb54c97b5a1e7881608ae08. Latest consensus state: current_term: 2 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:58.332844 7393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: LEADER
I20251029 16:33:58.333590 7392 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251029 16:33:58.334085 7394 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251029 16:33:58.338101 7394 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251029 16:33:58.339844 7394 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:33:58.340020 7394 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251029 16:33:58.341277 7392 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [LEADER]: Peer 3155690d4bb54c97b5a1e7881608ae08 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251029 16:33:58.341642 7395 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 1 LEADER]: Stepping down as leader of term 1
I20251029 16:33:58.341838 7395 raft_consensus.cc:740] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 29f9bab9976745478595681935778cac, State: Running, Role: LEADER
I20251029 16:33:58.342029 7394 catalog_manager.cc:1514] Loading token signing keys...
I20251029 16:33:58.342450 7395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:58.343168 7395 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 1 FOLLOWER]: Advancing to term 2
I20251029 16:33:58.343640 7394 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Loaded TSK: 0
I20251029 16:33:58.345191 7394 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251029 16:33:58.346570 7394 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251029 16:33:58.347555 7394 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
W20251029 16:33:58.360507 7390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:33:58.360865 7390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:33:58.361018 7390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:33:58.369781 7390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:33:58.369982 7390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.180.65
I20251029 16:33:58.380749 7390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.180.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.180.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.180.65
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:33:58.382519 7390 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:33:58.384663 7390 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 16:33:58.395411 7401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:33:58.395443 7400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:33:58.396591 7403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:33:58.397112 7390 server_base.cc:1047] running on GCE node
I20251029 16:33:58.397796 7390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 16:33:58.398839 7390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:33:58.400031 7390 hybrid_clock.cc:648] HybridClock initialized: now 1761755638399978 us; error 42 us; skew 500 ppm
I20251029 16:33:58.400404 7390 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:33:58.403035 7390 webserver.cc:492] Webserver started at http://127.6.180.65:36053/ using document root <none> and password file <none>
I20251029 16:33:58.403864 7390 fs_manager.cc:362] Metadata directory not provided
I20251029 16:33:58.404019 7390 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:33:58.404402 7390 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 16:33:58.407074 7390 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/data/instance:
uuid: "4a7cbc2353a042bba0a6103565a2828c"
format_stamp: "Formatted at 2025-10-29 16:33:58 on dist-test-slave-l1bh"
I20251029 16:33:58.408003 7390 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/wal/instance:
uuid: "4a7cbc2353a042bba0a6103565a2828c"
format_stamp: "Formatted at 2025-10-29 16:33:58 on dist-test-slave-l1bh"
I20251029 16:33:58.414469 7390 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20251029 16:33:58.418553 7409 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:33:58.420277 7390 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20251029 16:33:58.420468 7390 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/wal
uuid: "4a7cbc2353a042bba0a6103565a2828c"
format_stamp: "Formatted at 2025-10-29 16:33:58 on dist-test-slave-l1bh"
I20251029 16:33:58.420683 7390 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:33:58.450361 7390 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 16:33:58.451337 7390 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:33:58.451614 7390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 16:33:58.452694 7390 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251029 16:33:58.454643 7390 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251029 16:33:58.454754 7390 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251029 16:33:58.454879 7390 ts_tablet_manager.cc:616] Registered 0 tablets
I20251029 16:33:58.454973 7390 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251029 16:33:58.489362 7390 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.65:39551
I20251029 16:33:58.489459 7523 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.65:39551 every 8 connection(s)
I20251029 16:33:58.490922 7390 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/data/info.pb
I20251029 16:33:58.496627 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 7390
I20251029 16:33:58.496846 6865 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/wal/instance
I20251029 16:33:58.509140 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.180.66:0
--local_ip_for_outbound_sockets=127.6.180.66
--webserver_interface=127.6.180.66
--webserver_port=0
--tserver_master_addrs=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251029 16:33:58.527518 7342 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:33:58.527741 7342 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: loading cluster ID for follower catalog manager: success
I20251029 16:33:58.531879 7342 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: acquiring CA information for follower catalog manager: success
I20251029 16:33:58.535291 7342 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 16:33:58.542339 7525 heartbeater.cc:344] Connected to a master server at 127.6.180.126:36819
I20251029 16:33:58.542753 7525 heartbeater.cc:461] Registering TS with master...
I20251029 16:33:58.544072 7525 heartbeater.cc:507] Master 127.6.180.126:36819 requested a full tablet report, sending...
I20251029 16:33:58.544966 7526 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:33:58.545190 7526 heartbeater.cc:461] Registering TS with master...
I20251029 16:33:58.545694 7526 heartbeater.cc:507] Master 127.6.180.125:44441 requested a full tablet report, sending...
I20251029 16:33:58.546931 7139 ts_manager.cc:194] Registered new tserver with Master: 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551)
I20251029 16:33:58.547353 7209 ts_manager.cc:194] Registered new tserver with Master: 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551)
I20251029 16:33:58.549621 7524 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:33:58.549741 7524 heartbeater.cc:461] Registering TS with master...
I20251029 16:33:58.549705 7209 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.180.65:46169
I20251029 16:33:58.550269 7524 heartbeater.cc:507] Master 127.6.180.124:39271 requested a full tablet report, sending...
I20251029 16:33:58.553143 7356 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 16:33:58.554334 7279 ts_manager.cc:194] Registered new tserver with Master: 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551)
I20251029 16:33:58.833585 7148 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 2 FOLLOWER]: Refusing update from remote peer 3155690d4bb54c97b5a1e7881608ae08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251029 16:33:58.834861 7393 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251029 16:33:58.866636 7536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3155690d4bb54c97b5a1e7881608ae08. Latest consensus state: current_term: 2 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:58.867034 7536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:33:58.883129 7536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:58.883634 7536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:33:58.887207 7289 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 2 FOLLOWER]: Refusing update from remote peer 3155690d4bb54c97b5a1e7881608ae08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251029 16:33:58.888283 7393 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251029 16:33:58.896595 7538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:58.896937 7538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: LEADER
I20251029 16:33:58.913357 7542 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3155690d4bb54c97b5a1e7881608ae08)
I20251029 16:33:58.913637 7542 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:58.913465 7547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:58.915524 7217 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3155690d4bb54c97b5a1e7881608ae08" is_pre_election: true
I20251029 16:33:58.916181 7148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0442da13a653435dba87fc90d8e30d1f" is_pre_election: true
I20251029 16:33:58.914472 7543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3155690d4bb54c97b5a1e7881608ae08. Latest consensus state: current_term: 2 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:58.916602 7543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:33:58.914836 7542 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
I20251029 16:33:58.916953 7547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:33:58.916997 7264 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f9bab9976745478595681935778cac; no voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:33:58.917390 7547 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251029 16:33:58.937928 7532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:33:58.938292 7532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:33:58.938489 7532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:33:58.947382 7532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:33:58.947646 7532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.180.66
I20251029 16:33:58.958333 7532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.180.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.180.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.180.66
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:33:58.960317 7532 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:33:58.962370 7532 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 16:33:58.973227 7552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:33:58.974277 7553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:33:58.978042 7556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:33:58.978320 7532 server_base.cc:1047] running on GCE node
I20251029 16:33:58.978991 7532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 16:33:58.980099 7532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:33:58.981343 7532 hybrid_clock.cc:648] HybridClock initialized: now 1761755638981281 us; error 53 us; skew 500 ppm
I20251029 16:33:58.981670 7532 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:33:58.984336 7532 webserver.cc:492] Webserver started at http://127.6.180.66:32883/ using document root <none> and password file <none>
I20251029 16:33:58.985189 7532 fs_manager.cc:362] Metadata directory not provided
I20251029 16:33:58.985306 7532 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:33:58.985697 7532 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 16:33:58.988528 7532 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/data/instance:
uuid: "7ec98b5d57094792a1aff2a1df4e7a46"
format_stamp: "Formatted at 2025-10-29 16:33:58 on dist-test-slave-l1bh"
I20251029 16:33:58.989420 7532 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/wal/instance:
uuid: "7ec98b5d57094792a1aff2a1df4e7a46"
format_stamp: "Formatted at 2025-10-29 16:33:58 on dist-test-slave-l1bh"
I20251029 16:33:58.995692 7532 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20251029 16:33:58.999818 7561 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:33:59.001426 7532 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251029 16:33:59.001623 7532 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/wal
uuid: "7ec98b5d57094792a1aff2a1df4e7a46"
format_stamp: "Formatted at 2025-10-29 16:33:58 on dist-test-slave-l1bh"
I20251029 16:33:59.001850 7532 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:33:59.039261 7532 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 16:33:59.040328 7532 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:33:59.040694 7532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 16:33:59.041783 7532 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251029 16:33:59.043761 7532 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251029 16:33:59.043884 7532 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251029 16:33:59.044039 7532 ts_tablet_manager.cc:616] Registered 0 tablets
I20251029 16:33:59.044117 7532 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251029 16:33:59.052943 7526 heartbeater.cc:499] Master 127.6.180.125:44441 was elected leader, sending a full tablet report...
I20251029 16:33:59.079339 7532 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.66:33379
I20251029 16:33:59.079385 7674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.66:33379 every 8 connection(s)
I20251029 16:33:59.080946 7532 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/data/info.pb
I20251029 16:33:59.085974 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 7532
I20251029 16:33:59.086261 6865 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/wal/instance
I20251029 16:33:59.093613 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.180.67:0
--local_ip_for_outbound_sockets=127.6.180.67
--webserver_interface=127.6.180.67
--webserver_port=0
--tserver_master_addrs=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251029 16:33:59.102388 7677 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:33:59.102846 7677 heartbeater.cc:461] Registering TS with master...
I20251029 16:33:59.104518 7677 heartbeater.cc:507] Master 127.6.180.125:44441 requested a full tablet report, sending...
I20251029 16:33:59.106472 7209 ts_manager.cc:194] Registered new tserver with Master: 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:33:59.106660 7676 heartbeater.cc:344] Connected to a master server at 127.6.180.126:36819
I20251029 16:33:59.106848 7676 heartbeater.cc:461] Registering TS with master...
I20251029 16:33:59.107625 7676 heartbeater.cc:507] Master 127.6.180.126:36819 requested a full tablet report, sending...
I20251029 16:33:59.107790 7209 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.180.66:34107
I20251029 16:33:59.109270 7139 ts_manager.cc:194] Registered new tserver with Master: 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:33:59.116813 7675 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:33:59.116999 7675 heartbeater.cc:461] Registering TS with master...
I20251029 16:33:59.117405 7675 heartbeater.cc:507] Master 127.6.180.124:39271 requested a full tablet report, sending...
I20251029 16:33:59.118512 7279 ts_manager.cc:194] Registered new tserver with Master: 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:33:59.371013 7536 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3155690d4bb54c97b5a1e7881608ae08)
I20251029 16:33:59.371240 7536 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:59.372751 7536 leader_election.cc:290] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
I20251029 16:33:59.373526 7217 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0442da13a653435dba87fc90d8e30d1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3155690d4bb54c97b5a1e7881608ae08" is_pre_election: true
I20251029 16:33:59.373505 7289 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0442da13a653435dba87fc90d8e30d1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "29f9bab9976745478595681935778cac" is_pre_election: true
I20251029 16:33:59.374336 7126 leader_election.cc:304] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f; no voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:33:59.374734 7536 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251029 16:33:59.455116 7686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:59.455562 7686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: LEADER
W20251029 16:33:59.461822 7685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:33:59.462199 7685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:33:59.462350 7685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:33:59.470954 7685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:33:59.471271 7685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.180.67
I20251029 16:33:59.481568 7685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.180.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.180.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.180.67
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:33:59.483330 7685 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:33:59.485679 7685 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 16:33:59.498623 7694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:33:59.499343 7695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:33:59.500159 7697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:33:59.501222 7685 server_base.cc:1047] running on GCE node
I20251029 16:33:59.501799 7685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 16:33:59.502734 7685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:33:59.503978 7685 hybrid_clock.cc:648] HybridClock initialized: now 1761755639503909 us; error 45 us; skew 500 ppm
I20251029 16:33:59.504312 7685 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:33:59.506608 7685 webserver.cc:492] Webserver started at http://127.6.180.67:43363/ using document root <none> and password file <none>
I20251029 16:33:59.507364 7685 fs_manager.cc:362] Metadata directory not provided
I20251029 16:33:59.507455 7685 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:33:59.507884 7685 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 16:33:59.510538 7685 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/data/instance:
uuid: "b94ca437df08467bae40554a10450269"
format_stamp: "Formatted at 2025-10-29 16:33:59 on dist-test-slave-l1bh"
I20251029 16:33:59.511401 7685 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/wal/instance:
uuid: "b94ca437df08467bae40554a10450269"
format_stamp: "Formatted at 2025-10-29 16:33:59 on dist-test-slave-l1bh"
I20251029 16:33:59.517659 7685 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20251029 16:33:59.521430 7703 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:33:59.522843 7685 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251029 16:33:59.523030 7685 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/wal
uuid: "b94ca437df08467bae40554a10450269"
format_stamp: "Formatted at 2025-10-29 16:33:59 on dist-test-slave-l1bh"
I20251029 16:33:59.523257 7685 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:33:59.542892 7685 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 16:33:59.543869 7685 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:33:59.544206 7685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 16:33:59.545255 7685 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251029 16:33:59.547063 7685 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251029 16:33:59.547195 7685 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251029 16:33:59.547315 7685 ts_tablet_manager.cc:616] Registered 0 tablets
I20251029 16:33:59.547390 7685 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251029 16:33:59.583635 7685 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.67:38629
I20251029 16:33:59.583683 7816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.67:38629 every 8 connection(s)
I20251029 16:33:59.585284 7685 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/data/info.pb
I20251029 16:33:59.589813 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 7685
I20251029 16:33:59.590010 6865 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/wal/instance
I20251029 16:33:59.604565 7818 heartbeater.cc:344] Connected to a master server at 127.6.180.126:36819
I20251029 16:33:59.605099 7818 heartbeater.cc:461] Registering TS with master...
I20251029 16:33:59.606595 7818 heartbeater.cc:507] Master 127.6.180.126:36819 requested a full tablet report, sending...
I20251029 16:33:59.608711 7819 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:33:59.608918 7819 heartbeater.cc:461] Registering TS with master...
I20251029 16:33:59.609512 7819 heartbeater.cc:507] Master 127.6.180.125:44441 requested a full tablet report, sending...
I20251029 16:33:59.609844 7817 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:33:59.609972 7817 heartbeater.cc:461] Registering TS with master...
I20251029 16:33:59.610105 7139 ts_manager.cc:194] Registered new tserver with Master: b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:33:59.610502 7817 heartbeater.cc:507] Master 127.6.180.124:39271 requested a full tablet report, sending...
I20251029 16:33:59.611533 7209 ts_manager.cc:194] Registered new tserver with Master: b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:33:59.611763 7677 heartbeater.cc:499] Master 127.6.180.125:44441 was elected leader, sending a full tablet report...
I20251029 16:33:59.612020 7279 ts_manager.cc:194] Registered new tserver with Master: b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:33:59.612646 7209 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.180.67:47993
I20251029 16:33:59.618351 6865 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251029 16:33:59.651533 7209 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40712:
name: "default.Table_b513b87e8a1943eea3e24140ede4921d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:33:59.654794 7203 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40712:
name: "default.Table_4d4ea90e32c7496a91ee99d2ca5a1f01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:33:59.656016 7202 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40712:
name: "default.Table_0982a1f0341d4f3f9e7f27ed71278b4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:33:59.656628 7201 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40712:
name: "default.Table_6129e3888a3346669ab7c8982593f438"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:33:59.657558 7200 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40712:
name: "default.Table_207ddcecc1c54b149e04a4d6f8190acb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:33:59.653800 7205 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40712:
name: "default.Table_571d6fa0fe7e4782b8b0c692012fbd81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:33:59.654541 7204 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40712:
name: "default.Table_809b2f55b3594aa1aab4872f17b07c4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:33:59.650372 7208 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40712:
name: "default.Table_789f901f60d34caf9ddd2eb0e44c9dd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:33:59.652082 7207 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40712:
name: "default.Table_dfc244c581534b678f83267a7d10b58c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:33:59.653414 7206 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40712:
name: "default.Table_e6381c7d53654326b15714d30d7eba93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 16:33:59.666682 7209 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b513b87e8a1943eea3e24140ede4921d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 16:33:59.666878 7203 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 16:33:59.667649 7200 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_207ddcecc1c54b149e04a4d6f8190acb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 16:33:59.669152 7205 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_571d6fa0fe7e4782b8b0c692012fbd81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 16:33:59.669915 7201 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6129e3888a3346669ab7c8982593f438 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 16:33:59.672694 6865 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskoTseiL/build/asan/bin/kudu pid 7175
I20251029 16:33:59.674412 7239 generic_service.cc:196] Checking for leaks (request via RPC)
I20251029 16:33:59.808478 6865 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoTseiL/build/asan/bin/kudu with pid 7175
W20251029 16:33:59.842705 7827 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.180.125:44441)
W20251029 16:33:59.849951 7830 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 16:33:59.846363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.180.125:44441 (local address 127.0.0.1:40716)
1029 16:33:59.846744 (+ 381us) negotiation.cc:107] Waiting for socket to connect
1029 16:33:59.846768 (+ 24us) client_negotiation.cc:174] Beginning negotiation
1029 16:33:59.846975 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1029 16:33:59.848159 (+ 1184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":108}
I20251029 16:33:59.951588 7864 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3155690d4bb54c97b5a1e7881608ae08)
I20251029 16:33:59.952136 7864 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:59.956163 7289 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0442da13a653435dba87fc90d8e30d1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "29f9bab9976745478595681935778cac" is_pre_election: true
I20251029 16:33:59.956598 7289 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0442da13a653435dba87fc90d8e30d1f in term 2.
I20251029 16:33:59.957418 7124 leader_election.cc:304] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac; no voters:
I20251029 16:33:59.963197 7864 leader_election.cc:290] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
I20251029 16:33:59.963878 7864 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251029 16:33:59.964210 7864 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3155690d4bb54c97b5a1e7881608ae08)
I20251029 16:33:59.964428 7864 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 2 FOLLOWER]: Advancing to term 3
I20251029 16:33:59.968951 7864 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:59.970623 7289 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0442da13a653435dba87fc90d8e30d1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "29f9bab9976745478595681935778cac"
I20251029 16:33:59.970880 7289 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 2 FOLLOWER]: Advancing to term 3
I20251029 16:33:59.973881 7289 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0442da13a653435dba87fc90d8e30d1f in term 3.
I20251029 16:33:59.974578 7124 leader_election.cc:304] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac; no voters:
I20251029 16:33:59.974968 7867 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 3 FOLLOWER]: Leader election won for term 3
I20251029 16:33:59.975103 7864 leader_election.cc:290] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 3 election: Requested vote from peers 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:33:59.976925 7126 leader_election.cc:336] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
W20251029 16:33:59.978523 7126 leader_election.cc:336] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:33:59.979144 7867 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 3 LEADER]: Becoming Leader. State: Replica: 0442da13a653435dba87fc90d8e30d1f, State: Running, Role: LEADER
I20251029 16:33:59.979702 7867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:33:59.983776 7864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0442da13a653435dba87fc90d8e30d1f. Latest consensus state: current_term: 3 leader_uuid: "0442da13a653435dba87fc90d8e30d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:33:59.984014 7864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: This master's current role is: LEADER
I20251029 16:33:59.984755 7870 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251029 16:33:59.986747 7870 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251029 16:33:59.987834 7870 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:33:59.988023 7870 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251029 16:33:59.989629 7870 catalog_manager.cc:1514] Loading token signing keys...
I20251029 16:33:59.990933 7870 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: Loaded TSK: 0
I20251029 16:33:59.992342 7870 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251029 16:33:59.993763 7870 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251029 16:33:59.994874 7870 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251029 16:34:00.003674 7137 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_0982a1f0341d4f3f9e7f27ed71278b4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:00.004034 7138 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_6129e3888a3346669ab7c8982593f438"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:00.004109 7136 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_4d4ea90e32c7496a91ee99d2ca5a1f01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:00.003719 7135 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_b513b87e8a1943eea3e24140ede4921d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:00.005173 7133 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_dfc244c581534b678f83267a7d10b58c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:00.006981 7139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_207ddcecc1c54b149e04a4d6f8190acb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:00.006958 7131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_571d6fa0fe7e4782b8b0c692012fbd81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:00.005371 7134 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_789f901f60d34caf9ddd2eb0e44c9dd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:00.008857 7130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_809b2f55b3594aa1aab4872f17b07c4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:00.007979 7132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_e6381c7d53654326b15714d30d7eba93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 16:34:00.018244 7136 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 16:34:00.077077 7526 heartbeater.cc:646] Failed to heartbeat to 127.6.180.125:44441 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:00.115693 7819 heartbeater.cc:499] Master 127.6.180.125:44441 was elected leader, sending a full tablet report...
W20251029 16:34:00.116348 7677 heartbeater.cc:646] Failed to heartbeat to 127.6.180.125:44441 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
W20251029 16:34:00.117097 7819 heartbeater.cc:646] Failed to heartbeat to 127.6.180.125:44441 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:00.217356 7136 catalog_manager.cc:2174] created HMS catalog entry for table [id=d31229edb9834ddab88752bc8d367e80]
I20251029 16:34:00.222222 7289 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 3 FOLLOWER]: Refusing update from remote peer 0442da13a653435dba87fc90d8e30d1f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
W20251029 16:34:00.223066 7126 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f -> Peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Couldn't send request to peer 3155690d4bb54c97b5a1e7881608ae08. Status: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251029 16:34:00.223234 7864 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20251029 16:34:00.225440 7133 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_dfc244c581534b678f83267a7d10b58c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 16:34:00.225783 7131 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_571d6fa0fe7e4782b8b0c692012fbd81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 16:34:00.226074 7137 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0982a1f0341d4f3f9e7f27ed71278b4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 16:34:00.225783 7138 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6129e3888a3346669ab7c8982593f438 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 16:34:00.227063 7130 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_809b2f55b3594aa1aab4872f17b07c4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 16:34:00.228104 7135 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b513b87e8a1943eea3e24140ede4921d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 16:34:00.225440 7139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_207ddcecc1c54b149e04a4d6f8190acb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 16:34:00.229242 7132 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e6381c7d53654326b15714d30d7eba93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 16:34:00.230253 7134 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_789f901f60d34caf9ddd2eb0e44c9dd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 16:34:00.247262 7876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0442da13a653435dba87fc90d8e30d1f. Latest consensus state: current_term: 3 leader_uuid: "0442da13a653435dba87fc90d8e30d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:00.247623 7876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:00.255199 7876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0442da13a653435dba87fc90d8e30d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:00.255507 7876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:00.259878 7867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0442da13a653435dba87fc90d8e30d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:00.260242 7867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: This master's current role is: LEADER
I20251029 16:34:00.267457 7136 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55098:
table { table_name: "default.Table_4d4ea90e32c7496a91ee99d2ca5a1f01" } new_table_name: "default.Table_2ce7bbbb663d4ced9e2a662c79543a9e" modify_external_catalogs: true
I20251029 16:34:00.288858 7609 tablet_service.cc:1505] Processing CreateTablet for tablet a73fc5d428c14686946360a136cefef0 (DEFAULT_TABLE table=default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.290388 7131 catalog_manager.cc:2174] created HMS catalog entry for table [id=a3cb6d437da645b488adaae8b577b07e]
I20251029 16:34:00.291348 7609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a73fc5d428c14686946360a136cefef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.299384 7458 tablet_service.cc:1505] Processing CreateTablet for tablet a73fc5d428c14686946360a136cefef0 (DEFAULT_TABLE table=default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.301622 7458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a73fc5d428c14686946360a136cefef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.308308 7131 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55098:
table { table_name: "default.Table_571d6fa0fe7e4782b8b0c692012fbd81" } new_table_name: "default.Table_ca38a68ba7574beab9c638e82db00946" modify_external_catalogs: true
I20251029 16:34:00.314958 7751 tablet_service.cc:1505] Processing CreateTablet for tablet a73fc5d428c14686946360a136cefef0 (DEFAULT_TABLE table=default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.321801 7750 tablet_service.cc:1505] Processing CreateTablet for tablet b88321d957304d9e8056c09379b2d894 (DEFAULT_TABLE table=default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.322949 7750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88321d957304d9e8056c09379b2d894. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.324460 7751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a73fc5d428c14686946360a136cefef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.328685 7457 tablet_service.cc:1505] Processing CreateTablet for tablet b88321d957304d9e8056c09379b2d894 (DEFAULT_TABLE table=default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.329599 7457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88321d957304d9e8056c09379b2d894. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.347157 7888 tablet_bootstrap.cc:492] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269: Bootstrap starting.
I20251029 16:34:00.352905 7887 tablet_bootstrap.cc:492] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c: Bootstrap starting.
I20251029 16:34:00.353118 7608 tablet_service.cc:1505] Processing CreateTablet for tablet b88321d957304d9e8056c09379b2d894 (DEFAULT_TABLE table=default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.353854 7608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88321d957304d9e8056c09379b2d894. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.354714 7888 tablet_bootstrap.cc:654] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.356256 7888 log.cc:826] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269: Log is configured to *not* fsync() on all Append() calls
I20251029 16:34:00.363117 7886 tablet_bootstrap.cc:492] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46: Bootstrap starting.
I20251029 16:34:00.365979 7887 tablet_bootstrap.cc:654] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.367579 7887 log.cc:826] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c: Log is configured to *not* fsync() on all Append() calls
I20251029 16:34:00.369424 7886 tablet_bootstrap.cc:654] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.371049 7886 log.cc:826] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46: Log is configured to *not* fsync() on all Append() calls
I20251029 16:34:00.371163 7888 tablet_bootstrap.cc:492] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269: No bootstrap required, opened a new log
I20251029 16:34:00.371552 7888 ts_tablet_manager.cc:1403] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.000s
I20251029 16:34:00.374476 7887 tablet_bootstrap.cc:492] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c: No bootstrap required, opened a new log
I20251029 16:34:00.374727 7886 tablet_bootstrap.cc:492] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46: No bootstrap required, opened a new log
I20251029 16:34:00.374900 7887 ts_tablet_manager.cc:1403] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.000s
I20251029 16:34:00.375044 7886 ts_tablet_manager.cc:1403] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20251029 16:34:00.377645 7133 catalog_manager.cc:2174] created HMS catalog entry for table [id=c3822131c9e8475eb4d6bb49bf60e48f]
I20251029 16:34:00.382493 7886 raft_consensus.cc:359] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.383241 7886 raft_consensus.cc:385] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.383458 7886 raft_consensus.cc:740] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.384091 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.125:44441
--webserver_interface=127.6.180.125
--webserver_port=41231
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251029 16:34:00.384586 7886 consensus_queue.cc:260] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.385372 7133 master_service.cc:954] ReplaceTablet: received request to replace tablet b88321d957304d9e8056c09379b2d894 from {username='slave'} at 127.0.0.1:55098
I20251029 16:34:00.389288 7887 raft_consensus.cc:359] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.389875 7887 raft_consensus.cc:385] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.390010 7887 raft_consensus.cc:740] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.390851 7887 consensus_queue.cc:260] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.395030 7887 ts_tablet_manager.cc:1434] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c: Time spent starting tablet: real 0.020s user 0.009s sys 0.003s
I20251029 16:34:00.400794 7458 tablet_service.cc:1505] Processing CreateTablet for tablet 9cf599494ab84b11a77e2e9d322da969 (DEFAULT_TABLE table=default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.401319 7458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf599494ab84b11a77e2e9d322da969. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.401741 7609 tablet_service.cc:1505] Processing CreateTablet for tablet 9cf599494ab84b11a77e2e9d322da969 (DEFAULT_TABLE table=default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.402207 7609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf599494ab84b11a77e2e9d322da969. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.407634 7751 tablet_service.cc:1505] Processing CreateTablet for tablet 9cf599494ab84b11a77e2e9d322da969 (DEFAULT_TABLE table=default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.408205 7751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf599494ab84b11a77e2e9d322da969. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.410938 7525 heartbeater.cc:499] Master 127.6.180.126:36819 was elected leader, sending a full tablet report...
I20251029 16:34:00.413467 7887 tablet_bootstrap.cc:492] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c: Bootstrap starting.
I20251029 16:34:00.417645 7887 tablet_bootstrap.cc:654] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.422175 7888 raft_consensus.cc:359] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.422720 7888 raft_consensus.cc:385] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.422852 7888 raft_consensus.cc:740] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ca437df08467bae40554a10450269, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.424388 7133 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: Sending DeleteTablet for 3 replicas of tablet b88321d957304d9e8056c09379b2d894
I20251029 16:34:00.425187 7133 catalog_manager.cc:6719] ReplaceTablet: tablet b88321d957304d9e8056c09379b2d894 deleted and replaced by tablet af86f1498654491da54d0da20774edd5
I20251029 16:34:00.432361 7676 heartbeater.cc:499] Master 127.6.180.126:36819 was elected leader, sending a full tablet report...
I20251029 16:34:00.432471 7886 ts_tablet_manager.cc:1434] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent starting tablet: real 0.057s user 0.025s sys 0.020s
I20251029 16:34:00.434811 7609 tablet_service.cc:1552] Processing DeleteTablet for tablet b88321d957304d9e8056c09379b2d894 with delete_type TABLET_DATA_DELETED (replaced by tablet af86f1498654491da54d0da20774edd5) from {username='slave'} at 127.0.0.1:35034
I20251029 16:34:00.423840 7888 consensus_queue.cc:260] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.434552 7751 tablet_service.cc:1552] Processing DeleteTablet for tablet b88321d957304d9e8056c09379b2d894 with delete_type TABLET_DATA_DELETED (replaced by tablet af86f1498654491da54d0da20774edd5) from {username='slave'} at 127.0.0.1:49924
I20251029 16:34:00.440541 7458 tablet_service.cc:1552] Processing DeleteTablet for tablet b88321d957304d9e8056c09379b2d894 with delete_type TABLET_DATA_DELETED (replaced by tablet af86f1498654491da54d0da20774edd5) from {username='slave'} at 127.0.0.1:55870
I20251029 16:34:00.440541 7750 tablet_service.cc:1505] Processing CreateTablet for tablet af86f1498654491da54d0da20774edd5 (DEFAULT_TABLE table=default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.441077 7750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af86f1498654491da54d0da20774edd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.441485 7886 tablet_bootstrap.cc:492] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46: Bootstrap starting.
I20251029 16:34:00.443413 7886 tablet_bootstrap.cc:654] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.447245 7457 tablet_service.cc:1505] Processing CreateTablet for tablet af86f1498654491da54d0da20774edd5 (DEFAULT_TABLE table=default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.447501 7608 tablet_service.cc:1505] Processing CreateTablet for tablet af86f1498654491da54d0da20774edd5 (DEFAULT_TABLE table=default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.447741 7457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af86f1498654491da54d0da20774edd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.447891 7608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af86f1498654491da54d0da20774edd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.457072 7818 heartbeater.cc:499] Master 127.6.180.126:36819 was elected leader, sending a full tablet report...
W20251029 16:34:00.457044 7125 catalog_manager.cc:4965] TS 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551): delete failed for tablet b88321d957304d9e8056c09379b2d894 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b88321d957304d9e8056c09379b2d894 already in progress: creating tablet
I20251029 16:34:00.458513 7458 tablet_service.cc:1552] Processing DeleteTablet for tablet b88321d957304d9e8056c09379b2d894 with delete_type TABLET_DATA_DELETED (replaced by tablet af86f1498654491da54d0da20774edd5) from {username='slave'} at 127.0.0.1:55870
W20251029 16:34:00.459393 7125 catalog_manager.cc:4965] TS 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551): delete failed for tablet b88321d957304d9e8056c09379b2d894 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b88321d957304d9e8056c09379b2d894 already in progress: creating tablet
W20251029 16:34:00.461573 7123 catalog_manager.cc:4965] TS 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): delete failed for tablet b88321d957304d9e8056c09379b2d894 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b88321d957304d9e8056c09379b2d894 already in progress: creating tablet
I20251029 16:34:00.464951 7887 tablet_bootstrap.cc:492] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c: No bootstrap required, opened a new log
I20251029 16:34:00.465253 7887 ts_tablet_manager.cc:1403] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.000s
I20251029 16:34:00.466543 7887 raft_consensus.cc:359] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.466789 7887 raft_consensus.cc:385] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.466918 7887 raft_consensus.cc:740] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.467233 7887 consensus_queue.cc:260] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.469734 7133 master_service.cc:954] ReplaceTablet: received request to replace tablet a73fc5d428c14686946360a136cefef0 from {username='slave'} at 127.0.0.1:55098
I20251029 16:34:00.470168 7886 tablet_bootstrap.cc:492] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46: No bootstrap required, opened a new log
I20251029 16:34:00.470359 7886 ts_tablet_manager.cc:1403] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20251029 16:34:00.471493 7886 raft_consensus.cc:359] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.471701 7886 raft_consensus.cc:385] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.471788 7886 raft_consensus.cc:740] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.472122 7886 consensus_queue.cc:260] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
W20251029 16:34:00.472369 7123 catalog_manager.cc:4965] TS b94ca437df08467bae40554a10450269 (127.6.180.67:38629): delete failed for tablet b88321d957304d9e8056c09379b2d894 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b88321d957304d9e8056c09379b2d894 already in progress: creating tablet
I20251029 16:34:00.472693 7886 ts_tablet_manager.cc:1434] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 16:34:00.473012 7886 tablet_bootstrap.cc:492] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46: Bootstrap starting.
I20251029 16:34:00.474207 7887 ts_tablet_manager.cc:1434] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251029 16:34:00.474915 7886 tablet_bootstrap.cc:654] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.475351 7887 tablet_bootstrap.cc:492] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c: Bootstrap starting.
I20251029 16:34:00.477733 7887 tablet_bootstrap.cc:654] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.485297 7888 ts_tablet_manager.cc:1434] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269: Time spent starting tablet: real 0.099s user 0.000s sys 0.011s
I20251029 16:34:00.486083 7888 tablet_bootstrap.cc:492] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269: Bootstrap starting.
I20251029 16:34:00.488121 7888 tablet_bootstrap.cc:654] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.489243 7133 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: Sending DeleteTablet for 3 replicas of tablet a73fc5d428c14686946360a136cefef0
I20251029 16:34:00.490269 7458 tablet_service.cc:1552] Processing DeleteTablet for tablet a73fc5d428c14686946360a136cefef0 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f5493c0c43d4c4c9e782adaf7b7ede1) from {username='slave'} at 127.0.0.1:55870
I20251029 16:34:00.490546 7609 tablet_service.cc:1552] Processing DeleteTablet for tablet a73fc5d428c14686946360a136cefef0 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f5493c0c43d4c4c9e782adaf7b7ede1) from {username='slave'} at 127.0.0.1:35034
I20251029 16:34:00.490635 7898 tablet_replica.cc:333] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c: stopping tablet replica
I20251029 16:34:00.490881 7751 tablet_service.cc:1552] Processing DeleteTablet for tablet a73fc5d428c14686946360a136cefef0 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f5493c0c43d4c4c9e782adaf7b7ede1) from {username='slave'} at 127.0.0.1:49924
I20251029 16:34:00.491197 7898 raft_consensus.cc:2243] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:00.491422 7895 tablet_replica.cc:333] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46: stopping tablet replica
I20251029 16:34:00.491667 7898 raft_consensus.cc:2272] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:00.491888 7895 raft_consensus.cc:2243] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:00.492467 7133 catalog_manager.cc:6719] ReplaceTablet: tablet a73fc5d428c14686946360a136cefef0 deleted and replaced by tablet 8f5493c0c43d4c4c9e782adaf7b7ede1
I20251029 16:34:00.493541 7898 ts_tablet_manager.cc:1916] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:00.492354 7895 raft_consensus.cc:2272] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 16:34:00.496577 7527 tablet.cc:2378] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 16:34:00.497952 7458 tablet_service.cc:1552] Processing DeleteTablet for tablet b88321d957304d9e8056c09379b2d894 with delete_type TABLET_DATA_DELETED (replaced by tablet af86f1498654491da54d0da20774edd5) from {username='slave'} at 127.0.0.1:55870
I20251029 16:34:00.499315 7895 ts_tablet_manager.cc:1916] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:00.500078 7133 master_service.cc:954] ReplaceTablet: received request to replace tablet a73fc5d428c14686946360a136cefef0 from {username='slave'} at 127.0.0.1:55098
I20251029 16:34:00.501386 7133 master_service.cc:954] ReplaceTablet: received request to replace tablet a73fc5d428c14686946360a136cefef0 from {username='slave'} at 127.0.0.1:55098
I20251029 16:34:00.502380 7133 master_service.cc:954] ReplaceTablet: received request to replace tablet b88321d957304d9e8056c09379b2d894 from {username='slave'} at 127.0.0.1:55098
I20251029 16:34:00.503602 7133 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55098:
table { table_name: "default.Table_dfc244c581534b678f83267a7d10b58c" } new_table_name: "default.Table_35daf50c962249fd983f0e49e1f7960a" modify_external_catalogs: true
I20251029 16:34:00.509032 7130 catalog_manager.cc:2174] created HMS catalog entry for table [id=de52525dd5684e4da86d8f8c77ad80d6]
I20251029 16:34:00.511358 7458 tablet_service.cc:1505] Processing CreateTablet for tablet 8f5493c0c43d4c4c9e782adaf7b7ede1 (DEFAULT_TABLE table=default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.511857 7458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5493c0c43d4c4c9e782adaf7b7ede1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.513357 7898 ts_tablet_manager.cc:1929] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 16:34:00.513553 7898 log.cc:1199] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/wal/wals/a73fc5d428c14686946360a136cefef0
I20251029 16:34:00.514086 7898 ts_tablet_manager.cc:1950] T a73fc5d428c14686946360a136cefef0 P 4a7cbc2353a042bba0a6103565a2828c: Deleting consensus metadata
I20251029 16:34:00.514161 7609 tablet_service.cc:1552] Processing DeleteTablet for tablet b88321d957304d9e8056c09379b2d894 with delete_type TABLET_DATA_DELETED (replaced by tablet af86f1498654491da54d0da20774edd5) from {username='slave'} at 127.0.0.1:35034
I20251029 16:34:00.516944 7609 tablet_service.cc:1505] Processing CreateTablet for tablet 8f5493c0c43d4c4c9e782adaf7b7ede1 (DEFAULT_TABLE table=default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.517084 7751 tablet_service.cc:1505] Processing CreateTablet for tablet 8f5493c0c43d4c4c9e782adaf7b7ede1 (DEFAULT_TABLE table=default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.517452 7609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5493c0c43d4c4c9e782adaf7b7ede1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.517481 7751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5493c0c43d4c4c9e782adaf7b7ede1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.520733 7886 tablet_bootstrap.cc:492] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46: No bootstrap required, opened a new log
I20251029 16:34:00.520982 7886 ts_tablet_manager.cc:1403] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.001s
I20251029 16:34:00.521930 7886 raft_consensus.cc:359] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.522168 7886 raft_consensus.cc:385] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.522274 7886 raft_consensus.cc:740] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.525707 7895 ts_tablet_manager.cc:1929] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 16:34:00.525909 7895 log.cc:1199] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/wal/wals/a73fc5d428c14686946360a136cefef0
I20251029 16:34:00.525784 7130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_4bedb57357b24f78a75d1f362473ceff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:00.526391 7895 ts_tablet_manager.cc:1950] T a73fc5d428c14686946360a136cefef0 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting consensus metadata
I20251029 16:34:00.527755 7886 consensus_queue.cc:260] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.530731 7886 ts_tablet_manager.cc:1434] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20251029 16:34:00.531138 7886 tablet_bootstrap.cc:492] T af86f1498654491da54d0da20774edd5 P 7ec98b5d57094792a1aff2a1df4e7a46: Bootstrap starting.
I20251029 16:34:00.532764 7125 catalog_manager.cc:4985] TS 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551): tablet a73fc5d428c14686946360a136cefef0 (table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]) successfully deleted
I20251029 16:34:00.533284 7898 tablet_replica.cc:333] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c: stopping tablet replica
I20251029 16:34:00.538067 7898 raft_consensus.cc:2243] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:00.539829 7898 raft_consensus.cc:2272] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:00.540848 7898 ts_tablet_manager.cc:1916] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:00.541524 7886 tablet_bootstrap.cc:654] T af86f1498654491da54d0da20774edd5 P 7ec98b5d57094792a1aff2a1df4e7a46: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.542322 7888 tablet_bootstrap.cc:492] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269: No bootstrap required, opened a new log
I20251029 16:34:00.542505 7888 ts_tablet_manager.cc:1403] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.007s
I20251029 16:34:00.543615 7888 raft_consensus.cc:359] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.543843 7888 raft_consensus.cc:385] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.543972 7888 raft_consensus.cc:740] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ca437df08467bae40554a10450269, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.541270 7887 tablet_bootstrap.cc:492] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c: No bootstrap required, opened a new log
I20251029 16:34:00.544360 7895 tablet_replica.cc:333] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46: stopping tablet replica
I20251029 16:34:00.544366 7887 ts_tablet_manager.cc:1403] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c: Time spent bootstrapping tablet: real 0.069s user 0.002s sys 0.005s
I20251029 16:34:00.544725 7895 raft_consensus.cc:2243] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:00.545141 7895 raft_consensus.cc:2272] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:00.545199 7887 raft_consensus.cc:359] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.545416 7887 raft_consensus.cc:385] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.545506 7887 raft_consensus.cc:740] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.545804 7887 consensus_queue.cc:260] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.546144 7123 catalog_manager.cc:4985] TS 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): tablet a73fc5d428c14686946360a136cefef0 (table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]) successfully deleted
I20251029 16:34:00.546357 7458 tablet_service.cc:1505] Processing CreateTablet for tablet 1ddd091e76844d0cabcd2ea2ffe767cc (DEFAULT_TABLE table=default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.546855 7458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ddd091e76844d0cabcd2ea2ffe767cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.546996 7895 ts_tablet_manager.cc:1916] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:00.548362 7608 tablet_service.cc:1505] Processing CreateTablet for tablet 1ddd091e76844d0cabcd2ea2ffe767cc (DEFAULT_TABLE table=default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.548806 7608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ddd091e76844d0cabcd2ea2ffe767cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.550364 7897 tablet_replica.cc:333] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269: stopping tablet replica
I20251029 16:34:00.550913 7897 raft_consensus.cc:2243] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:00.551337 7897 raft_consensus.cc:2272] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:00.552351 7751 tablet_service.cc:1505] Processing CreateTablet for tablet 1ddd091e76844d0cabcd2ea2ffe767cc (DEFAULT_TABLE table=default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.552790 7751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ddd091e76844d0cabcd2ea2ffe767cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.560164 7887 ts_tablet_manager.cc:1434] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251029 16:34:00.561950 7897 ts_tablet_manager.cc:1916] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:00.563256 7887 tablet_bootstrap.cc:492] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c: Bootstrap starting.
I20251029 16:34:00.565102 7887 tablet_bootstrap.cc:654] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.564200 7888 consensus_queue.cc:260] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.569160 7888 ts_tablet_manager.cc:1434] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20251029 16:34:00.570600 7888 tablet_bootstrap.cc:492] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269: Bootstrap starting.
I20251029 16:34:00.572744 7888 tablet_bootstrap.cc:654] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269: Neither blocks nor log segments found. Creating new log.
W20251029 16:34:00.585038 7678 tablet.cc:2378] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251029 16:34:00.588479 7820 tablet.cc:2378] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 16:34:00.606063 7886 tablet_bootstrap.cc:492] T af86f1498654491da54d0da20774edd5 P 7ec98b5d57094792a1aff2a1df4e7a46: No bootstrap required, opened a new log
I20251029 16:34:00.606487 7886 ts_tablet_manager.cc:1403] T af86f1498654491da54d0da20774edd5 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent bootstrapping tablet: real 0.075s user 0.000s sys 0.004s
I20251029 16:34:00.607777 7887 tablet_bootstrap.cc:492] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c: No bootstrap required, opened a new log
I20251029 16:34:00.607636 7886 raft_consensus.cc:359] T af86f1498654491da54d0da20774edd5 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.608026 7887 ts_tablet_manager.cc:1403] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c: Time spent bootstrapping tablet: real 0.045s user 0.027s sys 0.014s
I20251029 16:34:00.608044 7886 raft_consensus.cc:385] T af86f1498654491da54d0da20774edd5 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.608278 7886 raft_consensus.cc:740] T af86f1498654491da54d0da20774edd5 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.608861 7886 consensus_queue.cc:260] T af86f1498654491da54d0da20774edd5 P 7ec98b5d57094792a1aff2a1df4e7a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.609321 7887 raft_consensus.cc:359] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.609558 7887 raft_consensus.cc:385] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.609648 7887 raft_consensus.cc:740] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.609934 7887 consensus_queue.cc:260] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.611909 7886 ts_tablet_manager.cc:1434] T af86f1498654491da54d0da20774edd5 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 16:34:00.613303 7886 tablet_bootstrap.cc:492] T 1ddd091e76844d0cabcd2ea2ffe767cc P 7ec98b5d57094792a1aff2a1df4e7a46: Bootstrap starting.
I20251029 16:34:00.615458 7886 tablet_bootstrap.cc:654] T 1ddd091e76844d0cabcd2ea2ffe767cc P 7ec98b5d57094792a1aff2a1df4e7a46: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.616997 7888 tablet_bootstrap.cc:492] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269: No bootstrap required, opened a new log
I20251029 16:34:00.617318 7888 ts_tablet_manager.cc:1403] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20251029 16:34:00.617580 7887 ts_tablet_manager.cc:1434] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251029 16:34:00.618281 7888 raft_consensus.cc:359] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.618608 7888 raft_consensus.cc:385] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.618738 7888 raft_consensus.cc:740] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ca437df08467bae40554a10450269, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.619082 7888 consensus_queue.cc:260] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.620916 7888 ts_tablet_manager.cc:1434] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 16:34:00.623644 7888 tablet_bootstrap.cc:492] T af86f1498654491da54d0da20774edd5 P b94ca437df08467bae40554a10450269: Bootstrap starting.
I20251029 16:34:00.625423 7887 tablet_bootstrap.cc:492] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c: Bootstrap starting.
I20251029 16:34:00.627367 7887 tablet_bootstrap.cc:654] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.629411 7895 ts_tablet_manager.cc:1929] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 16:34:00.629667 7895 log.cc:1199] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/wal/wals/b88321d957304d9e8056c09379b2d894
I20251029 16:34:00.630122 7895 ts_tablet_manager.cc:1950] T b88321d957304d9e8056c09379b2d894 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting consensus metadata
I20251029 16:34:00.631826 7898 ts_tablet_manager.cc:1929] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 16:34:00.632097 7898 log.cc:1199] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/wal/wals/b88321d957304d9e8056c09379b2d894
I20251029 16:34:00.632566 7898 ts_tablet_manager.cc:1950] T b88321d957304d9e8056c09379b2d894 P 4a7cbc2353a042bba0a6103565a2828c: Deleting consensus metadata
I20251029 16:34:00.634908 7125 catalog_manager.cc:4985] TS 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551): tablet b88321d957304d9e8056c09379b2d894 (table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]) successfully deleted
I20251029 16:34:00.638012 7887 tablet_bootstrap.cc:492] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c: No bootstrap required, opened a new log
I20251029 16:34:00.638239 7887 ts_tablet_manager.cc:1403] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20251029 16:34:00.639062 7887 raft_consensus.cc:359] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.639298 7887 raft_consensus.cc:385] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.639375 7887 raft_consensus.cc:740] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.639673 7887 consensus_queue.cc:260] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.643599 7887 ts_tablet_manager.cc:1434] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20251029 16:34:00.647264 7887 tablet_bootstrap.cc:492] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c: Bootstrap starting.
I20251029 16:34:00.647742 7888 tablet_bootstrap.cc:654] T af86f1498654491da54d0da20774edd5 P b94ca437df08467bae40554a10450269: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.649104 7887 tablet_bootstrap.cc:654] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.650012 7123 catalog_manager.cc:4985] TS 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): tablet b88321d957304d9e8056c09379b2d894 (table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]) successfully deleted
I20251029 16:34:00.651696 7887 tablet_bootstrap.cc:492] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c: No bootstrap required, opened a new log
I20251029 16:34:00.651804 7138 catalog_manager.cc:2174] created HMS catalog entry for table [id=54c21e8f41bb4f03be83c5cce599242c]
I20251029 16:34:00.651880 7887 ts_tablet_manager.cc:1403] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251029 16:34:00.652173 7897 ts_tablet_manager.cc:1929] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 16:34:00.652334 7897 log.cc:1199] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/wal/wals/a73fc5d428c14686946360a136cefef0
I20251029 16:34:00.652757 7897 ts_tablet_manager.cc:1950] T a73fc5d428c14686946360a136cefef0 P b94ca437df08467bae40554a10450269: Deleting consensus metadata
I20251029 16:34:00.652760 7887 raft_consensus.cc:359] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:00.652957 7887 raft_consensus.cc:385] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.653064 7887 raft_consensus.cc:740] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.653355 7887 consensus_queue.cc:260] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:00.654675 7123 catalog_manager.cc:4985] TS b94ca437df08467bae40554a10450269 (127.6.180.67:38629): tablet a73fc5d428c14686946360a136cefef0 (table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]) successfully deleted
I20251029 16:34:00.656880 7888 tablet_bootstrap.cc:492] T af86f1498654491da54d0da20774edd5 P b94ca437df08467bae40554a10450269: No bootstrap required, opened a new log
I20251029 16:34:00.657600 7887 ts_tablet_manager.cc:1434] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 16:34:00.658165 7886 tablet_bootstrap.cc:492] T 1ddd091e76844d0cabcd2ea2ffe767cc P 7ec98b5d57094792a1aff2a1df4e7a46: No bootstrap required, opened a new log
I20251029 16:34:00.658442 7886 ts_tablet_manager.cc:1403] T 1ddd091e76844d0cabcd2ea2ffe767cc P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20251029 16:34:00.659468 7886 raft_consensus.cc:359] T 1ddd091e76844d0cabcd2ea2ffe767cc P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:00.659758 7886 raft_consensus.cc:385] T 1ddd091e76844d0cabcd2ea2ffe767cc P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.659889 7886 raft_consensus.cc:740] T 1ddd091e76844d0cabcd2ea2ffe767cc P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.660279 7886 consensus_queue.cc:260] T 1ddd091e76844d0cabcd2ea2ffe767cc P 7ec98b5d57094792a1aff2a1df4e7a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:00.664211 7886 ts_tablet_manager.cc:1434] T 1ddd091e76844d0cabcd2ea2ffe767cc P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 16:34:00.664557 7888 ts_tablet_manager.cc:1403] T af86f1498654491da54d0da20774edd5 P b94ca437df08467bae40554a10450269: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20251029 16:34:00.668334 7886 tablet_bootstrap.cc:492] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46: Bootstrap starting.
I20251029 16:34:00.669497 7888 raft_consensus.cc:359] T af86f1498654491da54d0da20774edd5 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.673998 7888 raft_consensus.cc:385] T af86f1498654491da54d0da20774edd5 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.674276 7888 raft_consensus.cc:740] T af86f1498654491da54d0da20774edd5 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ca437df08467bae40554a10450269, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.674770 7458 tablet_service.cc:1505] Processing CreateTablet for tablet 41ad6aef032040f2933152a36de8d120 (DEFAULT_TABLE table=default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.675316 7458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ad6aef032040f2933152a36de8d120. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.675287 7888 consensus_queue.cc:260] T af86f1498654491da54d0da20774edd5 P b94ca437df08467bae40554a10450269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.679783 7886 tablet_bootstrap.cc:654] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.681428 7888 ts_tablet_manager.cc:1434] T af86f1498654491da54d0da20774edd5 P b94ca437df08467bae40554a10450269: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251029 16:34:00.681005 7609 tablet_service.cc:1505] Processing CreateTablet for tablet 41ad6aef032040f2933152a36de8d120 (DEFAULT_TABLE table=default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.682699 7609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ad6aef032040f2933152a36de8d120. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.681494 7751 tablet_service.cc:1505] Processing CreateTablet for tablet 41ad6aef032040f2933152a36de8d120 (DEFAULT_TABLE table=default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.682729 7888 tablet_bootstrap.cc:492] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269: Bootstrap starting.
I20251029 16:34:00.684321 7751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ad6aef032040f2933152a36de8d120. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.689803 7888 tablet_bootstrap.cc:654] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.709362 7750 tablet_service.cc:1552] Processing DeleteTablet for tablet b88321d957304d9e8056c09379b2d894 with delete_type TABLET_DATA_DELETED (replaced by tablet af86f1498654491da54d0da20774edd5) from {username='slave'} at 127.0.0.1:49924
I20251029 16:34:00.709713 7897 tablet_replica.cc:333] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269: stopping tablet replica
I20251029 16:34:00.709921 7897 raft_consensus.cc:2243] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:00.710110 7897 raft_consensus.cc:2272] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:00.711135 7897 ts_tablet_manager.cc:1916] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:00.711367 7886 tablet_bootstrap.cc:492] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46: No bootstrap required, opened a new log
I20251029 16:34:00.711540 7886 ts_tablet_manager.cc:1403] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.004s
I20251029 16:34:00.712459 7886 raft_consensus.cc:359] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.712687 7886 raft_consensus.cc:385] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.712776 7886 raft_consensus.cc:740] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.713074 7886 consensus_queue.cc:260] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.715615 7886 ts_tablet_manager.cc:1434] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251029 16:34:00.726262 7887 tablet_bootstrap.cc:492] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c: Bootstrap starting.
I20251029 16:34:00.728323 7887 tablet_bootstrap.cc:654] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.729599 7886 tablet_bootstrap.cc:492] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46: Bootstrap starting.
I20251029 16:34:00.729979 7888 tablet_bootstrap.cc:492] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269: No bootstrap required, opened a new log
I20251029 16:34:00.730984 7888 ts_tablet_manager.cc:1403] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.007s
I20251029 16:34:00.731519 7886 tablet_bootstrap.cc:654] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.732297 7888 raft_consensus.cc:359] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.732592 7888 raft_consensus.cc:385] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.732687 7888 raft_consensus.cc:740] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ca437df08467bae40554a10450269, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.733000 7888 consensus_queue.cc:260] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.738652 7888 ts_tablet_manager.cc:1434] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 16:34:00.740363 7887 tablet_bootstrap.cc:492] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c: No bootstrap required, opened a new log
I20251029 16:34:00.740597 7887 ts_tablet_manager.cc:1403] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20251029 16:34:00.741559 7887 raft_consensus.cc:359] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.741781 7887 raft_consensus.cc:385] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.741859 7887 raft_consensus.cc:740] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.742188 7887 consensus_queue.cc:260] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.743129 7888 tablet_bootstrap.cc:492] T 1ddd091e76844d0cabcd2ea2ffe767cc P b94ca437df08467bae40554a10450269: Bootstrap starting.
I20251029 16:34:00.744982 7887 ts_tablet_manager.cc:1434] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 16:34:00.749131 7888 tablet_bootstrap.cc:654] T 1ddd091e76844d0cabcd2ea2ffe767cc P b94ca437df08467bae40554a10450269: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.751636 7886 tablet_bootstrap.cc:492] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46: No bootstrap required, opened a new log
I20251029 16:34:00.751869 7886 ts_tablet_manager.cc:1403] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.009s
I20251029 16:34:00.752822 7886 raft_consensus.cc:359] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.753053 7886 raft_consensus.cc:385] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.753149 7886 raft_consensus.cc:740] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.754189 7897 ts_tablet_manager.cc:1929] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 16:34:00.754391 7897 log.cc:1199] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/wal/wals/b88321d957304d9e8056c09379b2d894
I20251029 16:34:00.754820 7897 ts_tablet_manager.cc:1950] T b88321d957304d9e8056c09379b2d894 P b94ca437df08467bae40554a10450269: Deleting consensus metadata
I20251029 16:34:00.756112 7123 catalog_manager.cc:4985] TS b94ca437df08467bae40554a10450269 (127.6.180.67:38629): tablet b88321d957304d9e8056c09379b2d894 (table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]) successfully deleted
I20251029 16:34:00.760924 7138 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_3a47f0a3057e46acaf74a5e33bffb30a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:00.762177 7751 tablet_service.cc:1552] Processing DeleteTablet for tablet b88321d957304d9e8056c09379b2d894 with delete_type TABLET_DATA_DELETED (replaced by tablet af86f1498654491da54d0da20774edd5) from {username='slave'} at 127.0.0.1:49924
W20251029 16:34:00.763595 7123 catalog_manager.cc:4952] TS b94ca437df08467bae40554a10450269 (127.6.180.67:38629): delete failed for tablet b88321d957304d9e8056c09379b2d894 because the tablet was not found. No further retry: Not found: Tablet not found: b88321d957304d9e8056c09379b2d894
I20251029 16:34:00.764231 7886 consensus_queue.cc:260] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.764986 7886 ts_tablet_manager.cc:1434] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20251029 16:34:00.768807 7888 tablet_bootstrap.cc:492] T 1ddd091e76844d0cabcd2ea2ffe767cc P b94ca437df08467bae40554a10450269: No bootstrap required, opened a new log
I20251029 16:34:00.769034 7888 ts_tablet_manager.cc:1403] T 1ddd091e76844d0cabcd2ea2ffe767cc P b94ca437df08467bae40554a10450269: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.003s
I20251029 16:34:00.770215 7888 raft_consensus.cc:359] T 1ddd091e76844d0cabcd2ea2ffe767cc P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:00.770463 7888 raft_consensus.cc:385] T 1ddd091e76844d0cabcd2ea2ffe767cc P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.770578 7888 raft_consensus.cc:740] T 1ddd091e76844d0cabcd2ea2ffe767cc P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ca437df08467bae40554a10450269, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.770924 7888 consensus_queue.cc:260] T 1ddd091e76844d0cabcd2ea2ffe767cc P b94ca437df08467bae40554a10450269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:00.774823 7888 ts_tablet_manager.cc:1434] T 1ddd091e76844d0cabcd2ea2ffe767cc P b94ca437df08467bae40554a10450269: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20251029 16:34:00.778537 7888 tablet_bootstrap.cc:492] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269: Bootstrap starting.
I20251029 16:34:00.780604 7888 tablet_bootstrap.cc:654] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:00.787770 7888 tablet_bootstrap.cc:492] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269: No bootstrap required, opened a new log
I20251029 16:34:00.788061 7888 ts_tablet_manager.cc:1403] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20251029 16:34:00.789168 7888 raft_consensus.cc:359] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.789436 7888 raft_consensus.cc:385] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:00.789527 7888 raft_consensus.cc:740] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ca437df08467bae40554a10450269, State: Initialized, Role: FOLLOWER
I20251029 16:34:00.789855 7888 consensus_queue.cc:260] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.794942 7888 ts_tablet_manager.cc:1434] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 16:34:00.877630 7894 raft_consensus.cc:493] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:00.877990 7894 raft_consensus.cc:515] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.880220 7894 leader_election.cc:290] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379), b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
16:34:00.893 [HMS - ERROR - pool-6-thread-1] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_b513b87e8a1943eea3e24140ede4921d already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20251029 16:34:00.896133 7135 catalog_manager.cc:2170] Already present: failed to create HMS catalog entry for table [id=43ebb16274ee4427b402709c6589cb17]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_b513b87e8a1943eea3e24140ede4921d already exists)
I20251029 16:34:00.902233 7629 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af86f1498654491da54d0da20774edd5" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" is_pre_election: true
I20251029 16:34:00.902835 7629 raft_consensus.cc:2468] T af86f1498654491da54d0da20774edd5 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a7cbc2353a042bba0a6103565a2828c in term 0.
I20251029 16:34:00.903806 7410 leader_election.cc:304] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c, 7ec98b5d57094792a1aff2a1df4e7a46; no voters:
I20251029 16:34:00.904371 7894 raft_consensus.cc:2804] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 16:34:00.904546 7894 raft_consensus.cc:493] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 16:34:00.904657 7894 raft_consensus.cc:3060] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:00.905289 7771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af86f1498654491da54d0da20774edd5" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94ca437df08467bae40554a10450269" is_pre_election: true
I20251029 16:34:00.905786 7771 raft_consensus.cc:2468] T af86f1498654491da54d0da20774edd5 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a7cbc2353a042bba0a6103565a2828c in term 0.
I20251029 16:34:00.907889 7894 raft_consensus.cc:515] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.909188 7629 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af86f1498654491da54d0da20774edd5" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46"
I20251029 16:34:00.909420 7629 raft_consensus.cc:3060] T af86f1498654491da54d0da20774edd5 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:00.912711 7629 raft_consensus.cc:2468] T af86f1498654491da54d0da20774edd5 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a7cbc2353a042bba0a6103565a2828c in term 1.
I20251029 16:34:00.913331 7410 leader_election.cc:304] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c, 7ec98b5d57094792a1aff2a1df4e7a46; no voters:
I20251029 16:34:00.913978 7894 leader_election.cc:290] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 election: Requested vote from peers 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379), b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:00.914280 7894 raft_consensus.cc:2804] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader election won for term 1
I20251029 16:34:00.914434 7894 raft_consensus.cc:697] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [term 1 LEADER]: Becoming Leader. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Running, Role: LEADER
I20251029 16:34:00.914636 7135 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55098:
table { table_name: "default.Table_809b2f55b3594aa1aab4872f17b07c4e" } new_table_name: "default.Table_feb27132a62f47388739326e4fb4118c" modify_external_catalogs: true
I20251029 16:34:00.914924 7894 consensus_queue.cc:237] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:00.929600 7771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af86f1498654491da54d0da20774edd5" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94ca437df08467bae40554a10450269"
I20251029 16:34:00.929875 7771 raft_consensus.cc:3060] T af86f1498654491da54d0da20774edd5 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:00.933284 7771 raft_consensus.cc:2468] T af86f1498654491da54d0da20774edd5 P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a7cbc2353a042bba0a6103565a2828c in term 1.
I20251029 16:34:00.936918 7894 raft_consensus.cc:493] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:00.937202 7894 raft_consensus.cc:515] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:00.937891 7894 leader_election.cc:290] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:00.939159 7771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ddd091e76844d0cabcd2ea2ffe767cc" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94ca437df08467bae40554a10450269" is_pre_election: true
I20251029 16:34:00.939450 7771 raft_consensus.cc:2468] T 1ddd091e76844d0cabcd2ea2ffe767cc P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a7cbc2353a042bba0a6103565a2828c in term 0.
I20251029 16:34:00.939518 7139 catalog_manager.cc:2174] created HMS catalog entry for table [id=bac0388305c24aa196e9a4e444e6915b]
I20251029 16:34:00.939870 7629 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ddd091e76844d0cabcd2ea2ffe767cc" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" is_pre_election: true
I20251029 16:34:00.940158 7629 raft_consensus.cc:2468] T 1ddd091e76844d0cabcd2ea2ffe767cc P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a7cbc2353a042bba0a6103565a2828c in term 0.
I20251029 16:34:00.940714 7410 leader_election.cc:304] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c, 7ec98b5d57094792a1aff2a1df4e7a46; no voters:
I20251029 16:34:00.941551 7894 raft_consensus.cc:2804] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 16:34:00.941668 7894 raft_consensus.cc:493] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 16:34:00.941773 7894 raft_consensus.cc:3060] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:00.944640 7894 raft_consensus.cc:515] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:00.945963 7771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ddd091e76844d0cabcd2ea2ffe767cc" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94ca437df08467bae40554a10450269"
I20251029 16:34:00.946236 7771 raft_consensus.cc:3060] T 1ddd091e76844d0cabcd2ea2ffe767cc P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:00.949168 7771 raft_consensus.cc:2468] T 1ddd091e76844d0cabcd2ea2ffe767cc P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a7cbc2353a042bba0a6103565a2828c in term 1.
I20251029 16:34:00.948940 7139 catalog_manager.cc:5649] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65). New cstate: current_term: 1 leader_uuid: "4a7cbc2353a042bba0a6103565a2828c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20251029 16:34:00.949901 7410 leader_election.cc:304] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c, b94ca437df08467bae40554a10450269; no voters:
I20251029 16:34:00.950281 7920 raft_consensus.cc:2804] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader election won for term 1
I20251029 16:34:00.950628 7894 leader_election.cc:290] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 election: Requested vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:00.951964 7629 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ddd091e76844d0cabcd2ea2ffe767cc" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46"
I20251029 16:34:00.952227 7629 raft_consensus.cc:3060] T 1ddd091e76844d0cabcd2ea2ffe767cc P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:00.955199 7629 raft_consensus.cc:2468] T 1ddd091e76844d0cabcd2ea2ffe767cc P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a7cbc2353a042bba0a6103565a2828c in term 1.
I20251029 16:34:00.966116 7920 raft_consensus.cc:697] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [term 1 LEADER]: Becoming Leader. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Running, Role: LEADER
I20251029 16:34:00.966699 7920 consensus_queue.cc:237] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:00.975663 7458 tablet_service.cc:1505] Processing CreateTablet for tablet e1ac79eb37644b3e8e02681941c20091 (DEFAULT_TABLE table=default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.976301 7609 tablet_service.cc:1505] Processing CreateTablet for tablet e1ac79eb37644b3e8e02681941c20091 (DEFAULT_TABLE table=default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.976819 7609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1ac79eb37644b3e8e02681941c20091. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.980427 7751 tablet_service.cc:1505] Processing CreateTablet for tablet e1ac79eb37644b3e8e02681941c20091 (DEFAULT_TABLE table=default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:00.981015 7751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1ac79eb37644b3e8e02681941c20091. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.985538 7458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1ac79eb37644b3e8e02681941c20091. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:00.988991 7132 catalog_manager.cc:2174] created HMS catalog entry for table [id=370cf332e83748ae807762a392d183ab]
I20251029 16:34:00.998059 7139 catalog_manager.cc:5649] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65). New cstate: current_term: 1 leader_uuid: "4a7cbc2353a042bba0a6103565a2828c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } health_report { overall_health: UNKNOWN } } }
I20251029 16:34:01.005481 7888 tablet_bootstrap.cc:492] T e1ac79eb37644b3e8e02681941c20091 P b94ca437df08467bae40554a10450269: Bootstrap starting.
I20251029 16:34:01.007578 7888 tablet_bootstrap.cc:654] T e1ac79eb37644b3e8e02681941c20091 P b94ca437df08467bae40554a10450269: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.012604 7132 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55098 (request call id 20) took 1009 ms. Trace:
I20251029 16:34:01.013443 7132 rpcz_store.cc:276] 1029 16:34:00.003405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 16:34:00.005062 (+ 1657us) service_pool.cc:225] Handling call
1029 16:34:00.229379 (+224317us) catalog_manager.cc:2101] Acquired catalog manager lock
1029 16:34:00.229831 (+ 452us) catalog_manager.cc:2147] Created new table and tablet info
1029 16:34:00.988920 (+759089us) catalog_manager.cc:2173] Created new table in HMS catalog
1029 16:34:01.012244 (+ 23324us) catalog_manager.cc:2207] Wrote table and tablets to system table
1029 16:34:01.012409 (+ 165us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1029 16:34:01.012554 (+ 145us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 16:34:00.989987 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 16:34:00.990129 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
1029 16:34:00.990150 (+ 21us) write_op.cc:435] Acquired schema lock
1029 16:34:00.990152 (+ 2us) tablet.cc:660] Decoding operations
1029 16:34:00.990255 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
1029 16:34:00.990270 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1029 16:34:00.990275 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1029 16:34:00.990361 (+ 86us) tablet.cc:699] Row locks acquired
1029 16:34:00.990363 (+ 2us) write_op.cc:260] PREPARE: finished
1029 16:34:00.990409 (+ 46us) write_op.cc:270] Start()
1029 16:34:00.990446 (+ 37us) write_op.cc:276] Timestamp: P: 1761755640990322 usec, L: 0
1029 16:34:00.990450 (+ 4us) op_driver.cc:348] REPLICATION: starting
1029 16:34:00.990655 (+ 205us) log.cc:844] Serialized 432 byte log entry
1029 16:34:01.008578 (+ 17923us) op_driver.cc:464] REPLICATION: finished
1029 16:34:01.011485 (+ 2907us) write_op.cc:301] APPLY: starting
1029 16:34:01.011510 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
1029 16:34:01.011564 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
1029 16:34:01.011604 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 16:34:01.011732 (+ 128us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 16:34:01.011775 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1029 16:34:01.011781 (+ 6us) write_op.cc:312] APPLY: finished
1029 16:34:01.011874 (+ 93us) log.cc:844] Serialized 28 byte log entry
1029 16:34:01.011988 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
1029 16:34:01.012011 (+ 23us) write_op.cc:454] Released schema lock
1029 16:34:01.012017 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1029 16:34:01.012119 (+ 102us) write_op.cc:489] Releasing partition, row and schema locks
1029 16:34:01.012122 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":709277,"Hive Metastore.run_cpu_time_us":358,"Hive Metastore.run_wall_time_us":49578,"spinlock_wait_cycles":1655040,"child_traces":[["op",{"apply.queue_time_us":2809,"apply.run_cpu_time_us":637,"apply.run_wall_time_us":637,"num_ops":2,"prepare.queue_time_us":149,"prepare.run_cpu_time_us":790,"prepare.run_wall_time_us":791,"raft.queue_time_us":77,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":18063}]]}
I20251029 16:34:01.015604 7132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_e4dfb9e8508e41669fcd08f8c79b24f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:01.019829 7886 tablet_bootstrap.cc:492] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46: Bootstrap starting.
I20251029 16:34:01.021759 7886 tablet_bootstrap.cc:654] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.023888 7887 tablet_bootstrap.cc:492] T e1ac79eb37644b3e8e02681941c20091 P 4a7cbc2353a042bba0a6103565a2828c: Bootstrap starting.
I20251029 16:34:01.025947 7887 tablet_bootstrap.cc:654] T e1ac79eb37644b3e8e02681941c20091 P 4a7cbc2353a042bba0a6103565a2828c: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.026289 7888 tablet_bootstrap.cc:492] T e1ac79eb37644b3e8e02681941c20091 P b94ca437df08467bae40554a10450269: No bootstrap required, opened a new log
I20251029 16:34:01.026491 7888 ts_tablet_manager.cc:1403] T e1ac79eb37644b3e8e02681941c20091 P b94ca437df08467bae40554a10450269: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.001s
I20251029 16:34:01.027323 7888 raft_consensus.cc:359] T e1ac79eb37644b3e8e02681941c20091 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.027544 7888 raft_consensus.cc:385] T e1ac79eb37644b3e8e02681941c20091 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.027529 7134 catalog_manager.cc:2174] created HMS catalog entry for table [id=6e4eb1073fdf46d19d4e3ae42a186ce0]
I20251029 16:34:01.027637 7888 raft_consensus.cc:740] T e1ac79eb37644b3e8e02681941c20091 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ca437df08467bae40554a10450269, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.027895 7888 consensus_queue.cc:260] T e1ac79eb37644b3e8e02681941c20091 P b94ca437df08467bae40554a10450269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.033988 7888 ts_tablet_manager.cc:1434] T e1ac79eb37644b3e8e02681941c20091 P b94ca437df08467bae40554a10450269: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20251029 16:34:01.042145 7139 master_service.cc:954] ReplaceTablet: received request to replace tablet 8f5493c0c43d4c4c9e782adaf7b7ede1 from {username='slave'} at 127.0.0.1:55098
I20251029 16:34:01.043489 7609 tablet_service.cc:1505] Processing CreateTablet for tablet a6d6b65d23ce4b84aab753d124d9e1cf (DEFAULT_TABLE table=default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.046317 7134 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55098 (request call id 18) took 1043 ms. Trace:
I20251029 16:34:01.046517 7134 rpcz_store.cc:276] 1029 16:34:00.003173 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 16:34:00.004852 (+ 1679us) service_pool.cc:225] Handling call
1029 16:34:00.230381 (+225529us) catalog_manager.cc:2101] Acquired catalog manager lock
1029 16:34:00.230840 (+ 459us) catalog_manager.cc:2147] Created new table and tablet info
1029 16:34:01.027467 (+796627us) catalog_manager.cc:2173] Created new table in HMS catalog
1029 16:34:01.045923 (+ 18456us) catalog_manager.cc:2207] Wrote table and tablets to system table
1029 16:34:01.046106 (+ 183us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1029 16:34:01.046256 (+ 150us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 16:34:01.029472 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 16:34:01.029634 (+ 162us) write_op.cc:432] Acquiring schema lock in shared mode
1029 16:34:01.029643 (+ 9us) write_op.cc:435] Acquired schema lock
1029 16:34:01.029647 (+ 4us) tablet.cc:660] Decoding operations
1029 16:34:01.029770 (+ 123us) write_op.cc:620] Acquiring the partition lock for write op
1029 16:34:01.029817 (+ 47us) write_op.cc:641] Partition lock acquired for write op
1029 16:34:01.029825 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
1029 16:34:01.029934 (+ 109us) tablet.cc:699] Row locks acquired
1029 16:34:01.029938 (+ 4us) write_op.cc:260] PREPARE: finished
1029 16:34:01.029982 (+ 44us) write_op.cc:270] Start()
1029 16:34:01.030027 (+ 45us) write_op.cc:276] Timestamp: P: 1761755641029961 usec, L: 0
1029 16:34:01.030032 (+ 5us) op_driver.cc:348] REPLICATION: starting
1029 16:34:01.030275 (+ 243us) log.cc:844] Serialized 432 byte log entry
1029 16:34:01.043900 (+ 13625us) op_driver.cc:464] REPLICATION: finished
1029 16:34:01.044651 (+ 751us) write_op.cc:301] APPLY: starting
1029 16:34:01.044672 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
1029 16:34:01.044724 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
1029 16:34:01.044767 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 16:34:01.044907 (+ 140us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 16:34:01.044958 (+ 51us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1029 16:34:01.044967 (+ 9us) write_op.cc:312] APPLY: finished
1029 16:34:01.045065 (+ 98us) log.cc:844] Serialized 28 byte log entry
1029 16:34:01.045147 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1029 16:34:01.045171 (+ 24us) write_op.cc:454] Released schema lock
1029 16:34:01.045179 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1029 16:34:01.045267 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
1029 16:34:01.045270 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":757915,"Hive Metastore.run_cpu_time_us":317,"Hive Metastore.run_wall_time_us":38250,"child_traces":[["op",{"apply.queue_time_us":646,"apply.run_cpu_time_us":612,"apply.run_wall_time_us":613,"num_ops":2,"prepare.queue_time_us":995,"prepare.run_cpu_time_us":913,"prepare.run_wall_time_us":912,"raft.queue_time_us":328,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":13830}]]}
I20251029 16:34:01.047672 7134 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55098:
table { table_name: "default.Table_6129e3888a3346669ab7c8982593f438" } new_table_name: "default.Table_323e8fdbeff34a12add0ad9e4c97accd" modify_external_catalogs: true
I20251029 16:34:01.054427 7920 raft_consensus.cc:493] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:01.054687 7920 raft_consensus.cc:515] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.055503 7920 leader_election.cc:290] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379), b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:01.056457 7629 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41ad6aef032040f2933152a36de8d120" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" is_pre_election: true
I20251029 16:34:01.056710 7629 raft_consensus.cc:2468] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a7cbc2353a042bba0a6103565a2828c in term 0.
I20251029 16:34:01.057133 7771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41ad6aef032040f2933152a36de8d120" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94ca437df08467bae40554a10450269" is_pre_election: true
I20251029 16:34:01.057376 7771 raft_consensus.cc:2468] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a7cbc2353a042bba0a6103565a2828c in term 0.
I20251029 16:34:01.057940 7410 leader_election.cc:304] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c, b94ca437df08467bae40554a10450269; no voters:
I20251029 16:34:01.058140 7139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: Sending DeleteTablet for 3 replicas of tablet 8f5493c0c43d4c4c9e782adaf7b7ede1
I20251029 16:34:01.058944 7920 raft_consensus.cc:2804] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 16:34:01.059074 7457 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f5493c0c43d4c4c9e782adaf7b7ede1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4233e0df604c4ff8a7e60528788d7d07) from {username='slave'} at 127.0.0.1:55870
I20251029 16:34:01.059422 7898 tablet_replica.cc:333] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c: stopping tablet replica
I20251029 16:34:01.059617 7898 raft_consensus.cc:2243] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:01.059505 7751 tablet_service.cc:1505] Processing CreateTablet for tablet a6d6b65d23ce4b84aab753d124d9e1cf (DEFAULT_TABLE table=default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.059831 7898 raft_consensus.cc:2272] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:01.059073 7920 raft_consensus.cc:493] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 16:34:01.060106 7751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d6b65d23ce4b84aab753d124d9e1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:01.060225 7920 raft_consensus.cc:3060] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.060882 7898 ts_tablet_manager.cc:1916] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:01.063359 7920 raft_consensus.cc:515] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.064704 7458 tablet_service.cc:1505] Processing CreateTablet for tablet a6d6b65d23ce4b84aab753d124d9e1cf (DEFAULT_TABLE table=default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.065178 7458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d6b65d23ce4b84aab753d124d9e1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:01.066031 7139 catalog_manager.cc:6719] ReplaceTablet: tablet 8f5493c0c43d4c4c9e782adaf7b7ede1 deleted and replaced by tablet 4233e0df604c4ff8a7e60528788d7d07
I20251029 16:34:01.066453 7750 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f5493c0c43d4c4c9e782adaf7b7ede1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4233e0df604c4ff8a7e60528788d7d07) from {username='slave'} at 127.0.0.1:49924
I20251029 16:34:01.066418 7629 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41ad6aef032040f2933152a36de8d120" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46"
I20251029 16:34:01.066653 7629 raft_consensus.cc:3060] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.066672 7771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41ad6aef032040f2933152a36de8d120" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94ca437df08467bae40554a10450269"
I20251029 16:34:01.066768 7897 tablet_replica.cc:333] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269: stopping tablet replica
I20251029 16:34:01.066880 7771 raft_consensus.cc:3060] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.066980 7897 raft_consensus.cc:2243] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:01.067179 7897 raft_consensus.cc:2272] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:01.064162 7609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d6b65d23ce4b84aab753d124d9e1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:01.069706 7629 raft_consensus.cc:2468] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a7cbc2353a042bba0a6103565a2828c in term 1.
I20251029 16:34:01.069738 7771 raft_consensus.cc:2468] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a7cbc2353a042bba0a6103565a2828c in term 1.
I20251029 16:34:01.070459 7608 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f5493c0c43d4c4c9e782adaf7b7ede1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4233e0df604c4ff8a7e60528788d7d07) from {username='slave'} at 127.0.0.1:35034
I20251029 16:34:01.070940 7895 tablet_replica.cc:333] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46: stopping tablet replica
I20251029 16:34:01.071161 7895 raft_consensus.cc:2243] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:01.071385 7895 raft_consensus.cc:2272] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:01.071422 7137 catalog_manager.cc:2174] created HMS catalog entry for table [id=1f98c7c5ebc94e9fa5111bc7845d9abe]
I20251029 16:34:01.072472 7897 ts_tablet_manager.cc:1916] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:01.072968 7898 ts_tablet_manager.cc:1929] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 16:34:01.073107 7898 log.cc:1199] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/wal/wals/8f5493c0c43d4c4c9e782adaf7b7ede1
I20251029 16:34:01.073501 7898 ts_tablet_manager.cc:1950] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 4a7cbc2353a042bba0a6103565a2828c: Deleting consensus metadata
I20251029 16:34:01.076364 7125 catalog_manager.cc:4985] TS 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551): tablet 8f5493c0c43d4c4c9e782adaf7b7ede1 (table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]) successfully deleted
I20251029 16:34:01.077247 7895 ts_tablet_manager.cc:1916] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:01.081307 7410 leader_election.cc:304] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c, 7ec98b5d57094792a1aff2a1df4e7a46; no voters:
I20251029 16:34:01.081737 7922 raft_consensus.cc:2804] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader election won for term 1
I20251029 16:34:01.081907 7922 raft_consensus.cc:697] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [term 1 LEADER]: Becoming Leader. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Running, Role: LEADER
I20251029 16:34:01.082257 7922 consensus_queue.cc:237] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.085716 7897 ts_tablet_manager.cc:1929] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 16:34:01.085876 7897 log.cc:1199] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/wal/wals/8f5493c0c43d4c4c9e782adaf7b7ede1
I20251029 16:34:01.086391 7920 leader_election.cc:290] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 election: Requested vote from peers 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379), b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:01.087467 7897 ts_tablet_manager.cc:1950] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P b94ca437df08467bae40554a10450269: Deleting consensus metadata
I20251029 16:34:01.089120 7123 catalog_manager.cc:4985] TS b94ca437df08467bae40554a10450269 (127.6.180.67:38629): tablet 8f5493c0c43d4c4c9e782adaf7b7ede1 (table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]) successfully deleted
I20251029 16:34:01.092957 7886 tablet_bootstrap.cc:492] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46: No bootstrap required, opened a new log
I20251029 16:34:01.093178 7886 ts_tablet_manager.cc:1403] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent bootstrapping tablet: real 0.073s user 0.011s sys 0.005s
I20251029 16:34:01.094197 7886 raft_consensus.cc:359] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.094412 7886 raft_consensus.cc:385] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.094522 7886 raft_consensus.cc:740] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.094797 7886 consensus_queue.cc:260] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.095583 7895 ts_tablet_manager.cc:1929] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 16:34:01.098374 7895 log.cc:1199] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/wal/wals/8f5493c0c43d4c4c9e782adaf7b7ede1
I20251029 16:34:01.098836 7895 ts_tablet_manager.cc:1950] T 8f5493c0c43d4c4c9e782adaf7b7ede1 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting consensus metadata
I20251029 16:34:01.101171 7886 ts_tablet_manager.cc:1434] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
F20251029 16:34:01.102775 7338 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1761755639, eventType=CREATE_TABLE, dbName=default, tableName=table_b513b87e8a1943eea3e24140ede4921d, message={"server":"","servicePrincipal":"","db":"default","table":"table_b513b87e8a1943eea3e24140ede4921d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b513b87e8a1943eea3e24140ede4921d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761755639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/warehouse/table_b513b87e8a1943eea3e24140ede4921d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b513b87e8a1943eea3e24140ede4921d\",\"kudu.table_id\":\"1398ac3c2f22427b96af2e9ff15e62d4\",\"transient_lastDdlTime\":\"1761755639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"415c0df9d5524c9a82ceee0f02442bac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761755640,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761755640, eventType=CREATE_TABLE, dbName=default, tableName=table_4d4ea90e32c7496a91ee99d2ca5a1f01, message={"server":"","servicePrincipal":"","db":"default","table":"table_4d4ea90e32c7496a91ee99d2ca5a1f01","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_4d4ea90e32c7496a91ee99d2ca5a1f01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761755640},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/warehouse/table_4d4ea90e32c7496a91ee99d2ca5a1f01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_4d4ea90e32c7496a91ee99d2ca5a1f01\",\"kudu.table_id\":\"d31229edb9834ddab88752bc8d367e80\",\"transient_lastDdlTime\":\"1761755640\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"415c0df9d5524c9a82ceee0f02442bac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761755640,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251029 16:34:01.103087 7123 catalog_manager.cc:4985] TS 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): tablet 8f5493c0c43d4c4c9e782adaf7b7ede1 (table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]) successfully deleted
I20251029 16:34:01.105472 7137 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55098 (request call id 15) took 1102 ms. Trace:
@ 0x7f3bae791dcd google::LogMessage::Fail() at ??:0
I20251029 16:34:01.105656 7137 rpcz_store.cc:276] 1029 16:34:00.002850 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 16:34:00.002965 (+ 115us) service_pool.cc:225] Handling call
1029 16:34:00.226208 (+223243us) catalog_manager.cc:2101] Acquired catalog manager lock
1029 16:34:00.226697 (+ 489us) catalog_manager.cc:2147] Created new table and tablet info
1029 16:34:01.071358 (+844661us) catalog_manager.cc:2173] Created new table in HMS catalog
1029 16:34:01.105081 (+ 33723us) catalog_manager.cc:2207] Wrote table and tablets to system table
1029 16:34:01.105249 (+ 168us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1029 16:34:01.105409 (+ 160us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 16:34:01.088249 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 16:34:01.088365 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
1029 16:34:01.088371 (+ 6us) write_op.cc:435] Acquired schema lock
1029 16:34:01.088373 (+ 2us) tablet.cc:660] Decoding operations
1029 16:34:01.088470 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
1029 16:34:01.088481 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1029 16:34:01.088485 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1029 16:34:01.088557 (+ 72us) tablet.cc:699] Row locks acquired
1029 16:34:01.088559 (+ 2us) write_op.cc:260] PREPARE: finished
1029 16:34:01.088594 (+ 35us) write_op.cc:270] Start()
1029 16:34:01.088627 (+ 33us) write_op.cc:276] Timestamp: P: 1761755641088571 usec, L: 0
1029 16:34:01.088629 (+ 2us) op_driver.cc:348] REPLICATION: starting
1029 16:34:01.088789 (+ 160us) log.cc:844] Serialized 432 byte log entry
1029 16:34:01.098695 (+ 9906us) op_driver.cc:464] REPLICATION: finished
1029 16:34:01.103443 (+ 4748us) write_op.cc:301] APPLY: starting
1029 16:34:01.103468 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
1029 16:34:01.103522 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
1029 16:34:01.103547 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 16:34:01.103724 (+ 177us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 16:34:01.103785 (+ 61us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1029 16:34:01.103793 (+ 8us) write_op.cc:312] APPLY: finished
1029 16:34:01.103895 (+ 102us) log.cc:844] Serialized 28 byte log entry
1029 16:34:01.104020 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
1029 16:34:01.104048 (+ 28us) write_op.cc:454] Released schema lock
1029 16:34:01.104055 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1029 16:34:01.104157 (+ 102us) write_op.cc:489] Releasing partition, row and schema locks
1029 16:34:01.104161 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":795614,"Hive Metastore.run_cpu_time_us":350,"Hive Metastore.run_wall_time_us":43982,"spinlock_wait_cycles":6400,"child_traces":[["op",{"apply.queue_time_us":4659,"apply.run_cpu_time_us":705,"apply.run_wall_time_us":706,"mutex_wait_us":15079,"num_ops":2,"prepare.queue_time_us":16046,"prepare.run_cpu_time_us":626,"prepare.run_wall_time_us":625,"raft.queue_time_us":669,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":10024}]]}
I20251029 16:34:01.106797 7137 master_service.cc:954] ReplaceTablet: received request to replace tablet 8f5493c0c43d4c4c9e782adaf7b7ede1 from {username='slave'} at 127.0.0.1:55098
I20251029 16:34:01.107434 7137 master_service.cc:954] ReplaceTablet: received request to replace tablet 9cf599494ab84b11a77e2e9d322da969 from {username='slave'} at 127.0.0.1:55098
I20251029 16:34:01.107331 7139 catalog_manager.cc:5649] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65). New cstate: current_term: 1 leader_uuid: "4a7cbc2353a042bba0a6103565a2828c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } health_report { overall_health: UNKNOWN } } }
@ 0x7f3bae795b93 google::LogMessage::SendToLog() at ??:0
I20251029 16:34:01.108244 7892 raft_consensus.cc:493] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:01.108441 7892 raft_consensus.cc:515] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
@ 0x7f3bae7917cc google::LogMessage::Flush() at ??:0
I20251029 16:34:01.110579 7892 leader_election.cc:290] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551), b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
@ 0x7f3bae791a11 google::LogMessage::~LogMessage() at ??:0
I20251029 16:34:01.112619 7888 tablet_bootstrap.cc:492] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269: Bootstrap starting.
I20251029 16:34:01.114894 7888 tablet_bootstrap.cc:654] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.116737 7887 tablet_bootstrap.cc:492] T e1ac79eb37644b3e8e02681941c20091 P 4a7cbc2353a042bba0a6103565a2828c: No bootstrap required, opened a new log
I20251029 16:34:01.116933 7887 ts_tablet_manager.cc:1403] T e1ac79eb37644b3e8e02681941c20091 P 4a7cbc2353a042bba0a6103565a2828c: Time spent bootstrapping tablet: real 0.093s user 0.012s sys 0.004s
I20251029 16:34:01.117118 7457 tablet_service.cc:1505] Processing CreateTablet for tablet 82b72b5cd6a1496eb09ac78395a285a1 (DEFAULT_TABLE table=default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.117663 7457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82b72b5cd6a1496eb09ac78395a285a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:01.117918 7887 raft_consensus.cc:359] T e1ac79eb37644b3e8e02681941c20091 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.118163 7887 raft_consensus.cc:385] T e1ac79eb37644b3e8e02681941c20091 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.118280 7887 raft_consensus.cc:740] T e1ac79eb37644b3e8e02681941c20091 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.118347 7608 tablet_service.cc:1505] Processing CreateTablet for tablet 82b72b5cd6a1496eb09ac78395a285a1 (DEFAULT_TABLE table=default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.118553 7887 consensus_queue.cc:260] T e1ac79eb37644b3e8e02681941c20091 P 4a7cbc2353a042bba0a6103565a2828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.118793 7608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82b72b5cd6a1496eb09ac78395a285a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:01.120918 7887 ts_tablet_manager.cc:1434] T e1ac79eb37644b3e8e02681941c20091 P 4a7cbc2353a042bba0a6103565a2828c: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20251029 16:34:01.127871 7887 tablet_bootstrap.cc:492] T 82b72b5cd6a1496eb09ac78395a285a1 P 4a7cbc2353a042bba0a6103565a2828c: Bootstrap starting.
I20251029 16:34:01.130023 7887 tablet_bootstrap.cc:654] T 82b72b5cd6a1496eb09ac78395a285a1 P 4a7cbc2353a042bba0a6103565a2828c: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.132649 7886 tablet_bootstrap.cc:492] T a6d6b65d23ce4b84aab753d124d9e1cf P 7ec98b5d57094792a1aff2a1df4e7a46: Bootstrap starting.
I20251029 16:34:01.134215 7139 master_service.cc:954] ReplaceTablet: received request to replace tablet 9cf599494ab84b11a77e2e9d322da969 from {username='slave'} at 127.0.0.1:55098
I20251029 16:34:01.134721 7886 tablet_bootstrap.cc:654] T a6d6b65d23ce4b84aab753d124d9e1cf P 7ec98b5d57094792a1aff2a1df4e7a46: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.138682 7751 tablet_service.cc:1505] Processing CreateTablet for tablet 82b72b5cd6a1496eb09ac78395a285a1 (DEFAULT_TABLE table=default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.139170 7751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82b72b5cd6a1496eb09ac78395a285a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:01.141264 7137 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: Sending DeleteTablet for 3 replicas of tablet 9cf599494ab84b11a77e2e9d322da969
I20251029 16:34:01.141817 7137 catalog_manager.cc:6719] ReplaceTablet: tablet 9cf599494ab84b11a77e2e9d322da969 deleted and replaced by tablet 28e1452396924512ac30d309085628ee
I20251029 16:34:01.142779 7750 tablet_service.cc:1552] Processing DeleteTablet for tablet 9cf599494ab84b11a77e2e9d322da969 with delete_type TABLET_DATA_DELETED (replaced by tablet 28e1452396924512ac30d309085628ee) from {username='slave'} at 127.0.0.1:49924
I20251029 16:34:01.143273 7137 master_service.cc:954] ReplaceTablet: received request to replace tablet 8f5493c0c43d4c4c9e782adaf7b7ede1 from {username='slave'} at 127.0.0.1:55098
I20251029 16:34:01.143352 7897 tablet_replica.cc:333] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269: stopping tablet replica
I20251029 16:34:01.143589 7897 raft_consensus.cc:2243] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:01.143796 7897 raft_consensus.cc:2272] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:01.144518 7897 ts_tablet_manager.cc:1916] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:01.143849 7137 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_ca14635728c14a7fbd4bce9c90ec208d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:01.145109 7771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1ac79eb37644b3e8e02681941c20091" candidate_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94ca437df08467bae40554a10450269" is_pre_election: true
I20251029 16:34:01.145359 7771 raft_consensus.cc:2468] T e1ac79eb37644b3e8e02681941c20091 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec98b5d57094792a1aff2a1df4e7a46 in term 0.
I20251029 16:34:01.145915 7139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55098:
name: "default.Table_70f60abb54374bb5a1a1475de5feec56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:01.145978 7562 leader_election.cc:304] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269; no voters:
I20251029 16:34:01.146448 7887 tablet_bootstrap.cc:492] T 82b72b5cd6a1496eb09ac78395a285a1 P 4a7cbc2353a042bba0a6103565a2828c: No bootstrap required, opened a new log
I20251029 16:34:01.146632 7887 ts_tablet_manager.cc:1403] T 82b72b5cd6a1496eb09ac78395a285a1 P 4a7cbc2353a042bba0a6103565a2828c: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20251029 16:34:01.147462 7887 raft_consensus.cc:359] T 82b72b5cd6a1496eb09ac78395a285a1 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.147692 7887 raft_consensus.cc:385] T 82b72b5cd6a1496eb09ac78395a285a1 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.147763 7887 raft_consensus.cc:740] T 82b72b5cd6a1496eb09ac78395a285a1 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.148133 7887 consensus_queue.cc:260] T 82b72b5cd6a1496eb09ac78395a285a1 P 4a7cbc2353a042bba0a6103565a2828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.148185 7892 raft_consensus.cc:2804] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 16:34:01.148340 7892 raft_consensus.cc:493] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 16:34:01.148455 7892 raft_consensus.cc:3060] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.150516 7887 ts_tablet_manager.cc:1434] T 82b72b5cd6a1496eb09ac78395a285a1 P 4a7cbc2353a042bba0a6103565a2828c: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20251029 16:34:01.152640 7887 tablet_bootstrap.cc:492] T a6d6b65d23ce4b84aab753d124d9e1cf P 4a7cbc2353a042bba0a6103565a2828c: Bootstrap starting.
I20251029 16:34:01.154919 7888 tablet_bootstrap.cc:492] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269: No bootstrap required, opened a new log
I20251029 16:34:01.155089 7888 ts_tablet_manager.cc:1403] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.007s
I20251029 16:34:01.155839 7888 raft_consensus.cc:359] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.156059 7888 raft_consensus.cc:385] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.156127 7888 raft_consensus.cc:740] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ca437df08467bae40554a10450269, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.156361 7888 consensus_queue.cc:260] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.156819 7888 ts_tablet_manager.cc:1434] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 16:34:01.157182 7888 tablet_bootstrap.cc:492] T 82b72b5cd6a1496eb09ac78395a285a1 P b94ca437df08467bae40554a10450269: Bootstrap starting.
I20251029 16:34:01.159019 7888 tablet_bootstrap.cc:654] T 82b72b5cd6a1496eb09ac78395a285a1 P b94ca437df08467bae40554a10450269: Neither blocks nor log segments found. Creating new log.
@ 0x7f3bc5153b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251029 16:34:01.164474 7892 raft_consensus.cc:515] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.168205 7458 tablet_service.cc:1552] Processing DeleteTablet for tablet 9cf599494ab84b11a77e2e9d322da969 with delete_type TABLET_DATA_DELETED (replaced by tablet 28e1452396924512ac30d309085628ee) from {username='slave'} at 127.0.0.1:55870
I20251029 16:34:01.168586 7898 tablet_replica.cc:333] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c: stopping tablet replica
I20251029 16:34:01.168645 7608 tablet_service.cc:1552] Processing DeleteTablet for tablet 9cf599494ab84b11a77e2e9d322da969 with delete_type TABLET_DATA_DELETED (replaced by tablet 28e1452396924512ac30d309085628ee) from {username='slave'} at 127.0.0.1:35034
I20251029 16:34:01.168816 7898 raft_consensus.cc:2243] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:01.168953 7895 tablet_replica.cc:333] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46: stopping tablet replica
I20251029 16:34:01.169015 7898 raft_consensus.cc:2272] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:01.169188 7895 raft_consensus.cc:2243] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:01.169365 7895 raft_consensus.cc:2272] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:01.169672 7898 ts_tablet_manager.cc:1916] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:01.169929 7895 ts_tablet_manager.cc:1916] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:01.172935 7609 tablet_service.cc:1505] Processing CreateTablet for tablet 6a78066ce12c4092b69722de857c7a2a (DEFAULT_TABLE table=default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.173566 7609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a78066ce12c4092b69722de857c7a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:01.176966 7898 ts_tablet_manager.cc:1929] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 16:34:01.177135 7898 log.cc:1199] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/wal/wals/9cf599494ab84b11a77e2e9d322da969
I20251029 16:34:01.177449 7898 ts_tablet_manager.cc:1950] T 9cf599494ab84b11a77e2e9d322da969 P 4a7cbc2353a042bba0a6103565a2828c: Deleting consensus metadata
I20251029 16:34:01.177623 7895 ts_tablet_manager.cc:1929] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 16:34:01.177801 7895 log.cc:1199] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/wal/wals/9cf599494ab84b11a77e2e9d322da969
I20251029 16:34:01.178244 7895 ts_tablet_manager.cc:1950] T 9cf599494ab84b11a77e2e9d322da969 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting consensus metadata
I20251029 16:34:01.178664 7125 catalog_manager.cc:4985] TS 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551): tablet 9cf599494ab84b11a77e2e9d322da969 (table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]) successfully deleted
I20251029 16:34:01.179414 7771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1ac79eb37644b3e8e02681941c20091" candidate_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94ca437df08467bae40554a10450269"
I20251029 16:34:01.179661 7771 raft_consensus.cc:3060] T e1ac79eb37644b3e8e02681941c20091 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.179777 7887 tablet_bootstrap.cc:654] T a6d6b65d23ce4b84aab753d124d9e1cf P 4a7cbc2353a042bba0a6103565a2828c: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.180721 7457 tablet_service.cc:1505] Processing CreateTablet for tablet 6a78066ce12c4092b69722de857c7a2a (DEFAULT_TABLE table=default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.180727 7751 tablet_service.cc:1505] Processing CreateTablet for tablet 6a78066ce12c4092b69722de857c7a2a (DEFAULT_TABLE table=default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.181160 7457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a78066ce12c4092b69722de857c7a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:01.181166 7751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a78066ce12c4092b69722de857c7a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:01.182837 7771 raft_consensus.cc:2468] T e1ac79eb37644b3e8e02681941c20091 P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec98b5d57094792a1aff2a1df4e7a46 in term 1.
I20251029 16:34:01.183413 7562 leader_election.cc:304] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269; no voters:
I20251029 16:34:01.185998 7886 tablet_bootstrap.cc:492] T a6d6b65d23ce4b84aab753d124d9e1cf P 7ec98b5d57094792a1aff2a1df4e7a46: No bootstrap required, opened a new log
I20251029 16:34:01.186378 7123 catalog_manager.cc:4985] TS 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): tablet 9cf599494ab84b11a77e2e9d322da969 (table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]) successfully deleted
I20251029 16:34:01.188679 7458 tablet_service.cc:1505] Processing CreateTablet for tablet 4233e0df604c4ff8a7e60528788d7d07 (DEFAULT_TABLE table=default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.188750 7750 tablet_service.cc:1505] Processing CreateTablet for tablet 4233e0df604c4ff8a7e60528788d7d07 (DEFAULT_TABLE table=default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.189167 7750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4233e0df604c4ff8a7e60528788d7d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:01.189175 7458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4233e0df604c4ff8a7e60528788d7d07. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f3bc5152411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251029 16:34:01.197320 7897 ts_tablet_manager.cc:1929] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 16:34:01.197541 7897 log.cc:1199] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/wal/wals/9cf599494ab84b11a77e2e9d322da969
I20251029 16:34:01.197976 7897 ts_tablet_manager.cc:1950] T 9cf599494ab84b11a77e2e9d322da969 P b94ca437df08467bae40554a10450269: Deleting consensus metadata
I20251029 16:34:01.199291 7123 catalog_manager.cc:4985] TS b94ca437df08467bae40554a10450269 (127.6.180.67:38629): tablet 9cf599494ab84b11a77e2e9d322da969 (table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]) successfully deleted
I20251029 16:34:01.201447 7888 tablet_bootstrap.cc:492] T 82b72b5cd6a1496eb09ac78395a285a1 P b94ca437df08467bae40554a10450269: No bootstrap required, opened a new log
I20251029 16:34:01.201856 7888 ts_tablet_manager.cc:1403] T 82b72b5cd6a1496eb09ac78395a285a1 P b94ca437df08467bae40554a10450269: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.001s
I20251029 16:34:01.202893 7888 raft_consensus.cc:359] T 82b72b5cd6a1496eb09ac78395a285a1 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.203094 7888 raft_consensus.cc:385] T 82b72b5cd6a1496eb09ac78395a285a1 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.203205 7888 raft_consensus.cc:740] T 82b72b5cd6a1496eb09ac78395a285a1 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ca437df08467bae40554a10450269, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.203516 7888 consensus_queue.cc:260] T 82b72b5cd6a1496eb09ac78395a285a1 P b94ca437df08467bae40554a10450269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.204154 7888 ts_tablet_manager.cc:1434] T 82b72b5cd6a1496eb09ac78395a285a1 P b94ca437df08467bae40554a10450269: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 16:34:01.204505 7888 tablet_bootstrap.cc:492] T 4233e0df604c4ff8a7e60528788d7d07 P b94ca437df08467bae40554a10450269: Bootstrap starting.
I20251029 16:34:01.206765 7888 tablet_bootstrap.cc:654] T 4233e0df604c4ff8a7e60528788d7d07 P b94ca437df08467bae40554a10450269: Neither blocks nor log segments found. Creating new log.
@ 0x7f3bc515a08c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251029 16:34:01.211951 7892 leader_election.cc:290] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [CANDIDATE]: Term 1 election: Requested vote from peers 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551), b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:01.217108 7886 ts_tablet_manager.cc:1403] T a6d6b65d23ce4b84aab753d124d9e1cf P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.001s
I20251029 16:34:01.217692 7458 tablet_service.cc:1505] Processing CreateTablet for tablet 28e1452396924512ac30d309085628ee (DEFAULT_TABLE table=default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.218307 7458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28e1452396924512ac30d309085628ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:01.221325 7608 tablet_service.cc:1505] Processing CreateTablet for tablet 4233e0df604c4ff8a7e60528788d7d07 (DEFAULT_TABLE table=default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.221534 7887 tablet_bootstrap.cc:492] T a6d6b65d23ce4b84aab753d124d9e1cf P 4a7cbc2353a042bba0a6103565a2828c: No bootstrap required, opened a new log
I20251029 16:34:01.221730 7887 ts_tablet_manager.cc:1403] T a6d6b65d23ce4b84aab753d124d9e1cf P 4a7cbc2353a042bba0a6103565a2828c: Time spent bootstrapping tablet: real 0.069s user 0.001s sys 0.008s
I20251029 16:34:01.221827 7608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4233e0df604c4ff8a7e60528788d7d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:01.222604 7887 raft_consensus.cc:359] T a6d6b65d23ce4b84aab753d124d9e1cf P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.222790 7887 raft_consensus.cc:385] T a6d6b65d23ce4b84aab753d124d9e1cf P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.222882 7887 raft_consensus.cc:740] T a6d6b65d23ce4b84aab753d124d9e1cf P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.223150 7887 consensus_queue.cc:260] T a6d6b65d23ce4b84aab753d124d9e1cf P 4a7cbc2353a042bba0a6103565a2828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
@ 0x7f3bc5159f09 std::_Function_handler<>::_M_invoke() at ??:0
I20251029 16:34:01.227506 7892 raft_consensus.cc:2804] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 16:34:01.228076 7887 ts_tablet_manager.cc:1434] T a6d6b65d23ce4b84aab753d124d9e1cf P 4a7cbc2353a042bba0a6103565a2828c: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 16:34:01.229233 7892 raft_consensus.cc:697] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Running, Role: LEADER
I20251029 16:34:01.229250 7478 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1ac79eb37644b3e8e02681941c20091" candidate_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7cbc2353a042bba0a6103565a2828c" is_pre_election: true
I20251029 16:34:01.229700 7478 raft_consensus.cc:2468] T e1ac79eb37644b3e8e02681941c20091 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec98b5d57094792a1aff2a1df4e7a46 in term 0.
I20251029 16:34:01.229895 7892 consensus_queue.cc:237] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.230213 7477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1ac79eb37644b3e8e02681941c20091" candidate_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7cbc2353a042bba0a6103565a2828c"
I20251029 16:34:01.230465 7477 raft_consensus.cc:3060] T e1ac79eb37644b3e8e02681941c20091 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.231873 7887 tablet_bootstrap.cc:492] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c: Bootstrap starting.
@ 0xa8867d std::function<>::operator()() at ??:?
I20251029 16:34:01.232863 7886 raft_consensus.cc:359] T a6d6b65d23ce4b84aab753d124d9e1cf P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.233098 7886 raft_consensus.cc:385] T a6d6b65d23ce4b84aab753d124d9e1cf P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.233184 7886 raft_consensus.cc:740] T a6d6b65d23ce4b84aab753d124d9e1cf P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.233457 7886 consensus_queue.cc:260] T a6d6b65d23ce4b84aab753d124d9e1cf P 7ec98b5d57094792a1aff2a1df4e7a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.233906 7887 tablet_bootstrap.cc:654] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.234769 7477 raft_consensus.cc:2468] T e1ac79eb37644b3e8e02681941c20091 P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec98b5d57094792a1aff2a1df4e7a46 in term 1.
I20251029 16:34:01.236574 7607 tablet_service.cc:1505] Processing CreateTablet for tablet 28e1452396924512ac30d309085628ee (DEFAULT_TABLE table=default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.237229 7607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28e1452396924512ac30d309085628ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:01.248292 7751 tablet_service.cc:1505] Processing CreateTablet for tablet 28e1452396924512ac30d309085628ee (DEFAULT_TABLE table=default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:01.248970 7751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28e1452396924512ac30d309085628ee. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f3bb1505ee9 kudu::Thread::SuperviseThread() at ??:0
W20251029 16:34:01.253221 7562 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.180.67:38629, user_credentials={real_user=slave}} blocked reactor thread for 69779.8us
I20251029 16:34:01.256327 7887 tablet_bootstrap.cc:492] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c: No bootstrap required, opened a new log
I20251029 16:34:01.256507 7887 ts_tablet_manager.cc:1403] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20251029 16:34:01.257359 7887 raft_consensus.cc:359] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.257576 7887 raft_consensus.cc:385] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.257651 7887 raft_consensus.cc:740] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.257951 7887 consensus_queue.cc:260] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.258901 7887 ts_tablet_manager.cc:1434] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 16:34:01.259249 7887 tablet_bootstrap.cc:492] T 28e1452396924512ac30d309085628ee P 4a7cbc2353a042bba0a6103565a2828c: Bootstrap starting.
I20251029 16:34:01.261145 7888 tablet_bootstrap.cc:492] T 4233e0df604c4ff8a7e60528788d7d07 P b94ca437df08467bae40554a10450269: No bootstrap required, opened a new log
I20251029 16:34:01.261348 7888 ts_tablet_manager.cc:1403] T 4233e0df604c4ff8a7e60528788d7d07 P b94ca437df08467bae40554a10450269: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.009s
I20251029 16:34:01.261535 7887 tablet_bootstrap.cc:654] T 28e1452396924512ac30d309085628ee P 4a7cbc2353a042bba0a6103565a2828c: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.262495 7888 raft_consensus.cc:359] T 4233e0df604c4ff8a7e60528788d7d07 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.262701 7888 raft_consensus.cc:385] T 4233e0df604c4ff8a7e60528788d7d07 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.262789 7888 raft_consensus.cc:740] T 4233e0df604c4ff8a7e60528788d7d07 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ca437df08467bae40554a10450269, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.263060 7888 consensus_queue.cc:260] T 4233e0df604c4ff8a7e60528788d7d07 P b94ca437df08467bae40554a10450269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.263675 7888 ts_tablet_manager.cc:1434] T 4233e0df604c4ff8a7e60528788d7d07 P b94ca437df08467bae40554a10450269: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 16:34:01.264149 7888 tablet_bootstrap.cc:492] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269: Bootstrap starting.
I20251029 16:34:01.266155 7888 tablet_bootstrap.cc:654] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.266655 7922 raft_consensus.cc:493] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:01.266908 7922 raft_consensus.cc:515] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
@ 0x7f3bb4b106db start_thread at ??:0
I20251029 16:34:01.267796 7922 leader_election.cc:290] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379), b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:01.268810 7771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4233e0df604c4ff8a7e60528788d7d07" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94ca437df08467bae40554a10450269" is_pre_election: true
@ 0x7f3bad64271f clone at ??:0
I20251029 16:34:01.269076 7771 raft_consensus.cc:2468] T 4233e0df604c4ff8a7e60528788d7d07 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a7cbc2353a042bba0a6103565a2828c in term 0.
I20251029 16:34:01.269894 7410 leader_election.cc:304] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c, b94ca437df08467bae40554a10450269; no voters:
I20251029 16:34:01.270505 7896 raft_consensus.cc:493] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:01.270709 7896 raft_consensus.cc:515] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.272377 7886 ts_tablet_manager.cc:1434] T a6d6b65d23ce4b84aab753d124d9e1cf P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent starting tablet: real 0.055s user 0.000s sys 0.009s
I20251029 16:34:01.272924 7886 tablet_bootstrap.cc:492] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46: Bootstrap starting.
I20251029 16:34:01.273259 7896 leader_election.cc:290] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:01.274740 7886 tablet_bootstrap.cc:654] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.279134 7888 tablet_bootstrap.cc:492] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269: No bootstrap required, opened a new log
I20251029 16:34:01.279337 7888 ts_tablet_manager.cc:1403] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20251029 16:34:01.280320 7888 raft_consensus.cc:359] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.280562 7888 raft_consensus.cc:385] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.280659 7888 raft_consensus.cc:740] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ca437df08467bae40554a10450269, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.280902 7888 consensus_queue.cc:260] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.281533 7888 ts_tablet_manager.cc:1434] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 16:34:01.281713 7629 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4233e0df604c4ff8a7e60528788d7d07" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" is_pre_election: true
I20251029 16:34:01.281927 7888 tablet_bootstrap.cc:492] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269: Bootstrap starting.
I20251029 16:34:01.289033 7888 tablet_bootstrap.cc:654] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269: Neither blocks nor log segments found. Creating new log.
W20251029 16:34:01.289615 7410 leader_election.cc:343] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Illegal state: must be running to vote when last-logged opid is not known
I20251029 16:34:01.290338 7886 tablet_bootstrap.cc:492] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46: No bootstrap required, opened a new log
I20251029 16:34:01.290488 7886 ts_tablet_manager.cc:1403] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20251029 16:34:01.290855 7922 raft_consensus.cc:2804] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 16:34:01.290989 7922 raft_consensus.cc:493] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 16:34:01.291116 7922 raft_consensus.cc:3060] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.291334 7886 raft_consensus.cc:359] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.291522 7886 raft_consensus.cc:385] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.291603 7886 raft_consensus.cc:740] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.291818 7886 consensus_queue.cc:260] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.301645 7886 ts_tablet_manager.cc:1434] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251029 16:34:01.301981 7886 tablet_bootstrap.cc:492] T 28e1452396924512ac30d309085628ee P 7ec98b5d57094792a1aff2a1df4e7a46: Bootstrap starting.
I20251029 16:34:01.304028 7886 tablet_bootstrap.cc:654] T 28e1452396924512ac30d309085628ee P 7ec98b5d57094792a1aff2a1df4e7a46: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.315573 7886 tablet_bootstrap.cc:492] T 28e1452396924512ac30d309085628ee P 7ec98b5d57094792a1aff2a1df4e7a46: No bootstrap required, opened a new log
I20251029 16:34:01.315778 7886 ts_tablet_manager.cc:1403] T 28e1452396924512ac30d309085628ee P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20251029 16:34:01.316690 7886 raft_consensus.cc:359] T 28e1452396924512ac30d309085628ee P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.316920 7886 raft_consensus.cc:385] T 28e1452396924512ac30d309085628ee P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.316988 7886 raft_consensus.cc:740] T 28e1452396924512ac30d309085628ee P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.317257 7886 consensus_queue.cc:260] T 28e1452396924512ac30d309085628ee P 7ec98b5d57094792a1aff2a1df4e7a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.317940 7886 ts_tablet_manager.cc:1434] T 28e1452396924512ac30d309085628ee P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251029 16:34:01.318341 7886 tablet_bootstrap.cc:492] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46: Bootstrap starting.
I20251029 16:34:01.318753 7477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6d6b65d23ce4b84aab753d124d9e1cf" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7cbc2353a042bba0a6103565a2828c" is_pre_election: true
I20251029 16:34:01.318993 7477 raft_consensus.cc:2468] T a6d6b65d23ce4b84aab753d124d9e1cf P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b94ca437df08467bae40554a10450269 in term 0.
I20251029 16:34:01.319710 7706 leader_election.cc:304] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c, b94ca437df08467bae40554a10450269; no voters:
I20251029 16:34:01.320209 7886 tablet_bootstrap.cc:654] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.321630 7896 raft_consensus.cc:2804] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 16:34:01.321810 7896 raft_consensus.cc:493] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 16:34:01.321929 7896 raft_consensus.cc:3060] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.295447 7922 raft_consensus.cc:515] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.337703 7629 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4233e0df604c4ff8a7e60528788d7d07" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46"
W20251029 16:34:01.338626 7410 leader_election.cc:343] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Illegal state: must be running to vote when last-logged opid is not known
I20251029 16:34:01.342242 7771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4233e0df604c4ff8a7e60528788d7d07" candidate_uuid: "4a7cbc2353a042bba0a6103565a2828c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94ca437df08467bae40554a10450269"
I20251029 16:34:01.342556 7771 raft_consensus.cc:3060] T 4233e0df604c4ff8a7e60528788d7d07 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.343325 7922 leader_election.cc:290] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 election: Requested vote from peers 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379), b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:01.350484 7771 raft_consensus.cc:2468] T 4233e0df604c4ff8a7e60528788d7d07 P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a7cbc2353a042bba0a6103565a2828c in term 1.
I20251029 16:34:01.352669 7410 leader_election.cc:304] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c, b94ca437df08467bae40554a10450269; no voters: 7ec98b5d57094792a1aff2a1df4e7a46
I20251029 16:34:01.356475 7922 raft_consensus.cc:2804] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader election won for term 1
I20251029 16:34:01.356770 7922 raft_consensus.cc:697] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [term 1 LEADER]: Becoming Leader. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Running, Role: LEADER
I20251029 16:34:01.358595 7896 raft_consensus.cc:515] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.360291 7477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6d6b65d23ce4b84aab753d124d9e1cf" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7cbc2353a042bba0a6103565a2828c"
I20251029 16:34:01.360620 7477 raft_consensus.cc:3060] T a6d6b65d23ce4b84aab753d124d9e1cf P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.365223 7477 raft_consensus.cc:2468] T a6d6b65d23ce4b84aab753d124d9e1cf P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94ca437df08467bae40554a10450269 in term 1.
I20251029 16:34:01.365972 7706 leader_election.cc:304] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c, b94ca437df08467bae40554a10450269; no voters:
I20251029 16:34:01.368445 7954 raft_consensus.cc:2804] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 16:34:01.368692 7922 consensus_queue.cc:237] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.369180 7954 raft_consensus.cc:697] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [term 1 LEADER]: Becoming Leader. State: Replica: b94ca437df08467bae40554a10450269, State: Running, Role: LEADER
I20251029 16:34:01.369724 7954 consensus_queue.cc:237] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.377295 7628 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6d6b65d23ce4b84aab753d124d9e1cf" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46"
I20251029 16:34:01.377782 7628 raft_consensus.cc:3060] T a6d6b65d23ce4b84aab753d124d9e1cf P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.380216 7629 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6d6b65d23ce4b84aab753d124d9e1cf" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" is_pre_election: true
I20251029 16:34:01.383405 7628 raft_consensus.cc:2468] T a6d6b65d23ce4b84aab753d124d9e1cf P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94ca437df08467bae40554a10450269 in term 1.
I20251029 16:34:01.386873 7896 leader_election.cc:290] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 election: Requested vote from peers 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:01.391896 7892 raft_consensus.cc:493] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:01.392133 7892 raft_consensus.cc:515] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.392951 7892 leader_election.cc:290] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551), b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:01.394699 7477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82b72b5cd6a1496eb09ac78395a285a1" candidate_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7cbc2353a042bba0a6103565a2828c" is_pre_election: true
I20251029 16:34:01.395095 7477 raft_consensus.cc:2468] T 82b72b5cd6a1496eb09ac78395a285a1 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec98b5d57094792a1aff2a1df4e7a46 in term 0.
I20251029 16:34:01.395789 7564 leader_election.cc:304] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c, 7ec98b5d57094792a1aff2a1df4e7a46; no voters:
I20251029 16:34:01.396205 7892 raft_consensus.cc:2804] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 16:34:01.396310 7892 raft_consensus.cc:493] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 16:34:01.396405 7892 raft_consensus.cc:3060] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.399524 7892 raft_consensus.cc:515] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.402048 7892 leader_election.cc:290] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [CANDIDATE]: Term 1 election: Requested vote from peers 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551), b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:01.401976 7770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82b72b5cd6a1496eb09ac78395a285a1" candidate_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94ca437df08467bae40554a10450269"
I20251029 16:34:01.402387 7770 raft_consensus.cc:3060] T 82b72b5cd6a1496eb09ac78395a285a1 P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.402878 7771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82b72b5cd6a1496eb09ac78395a285a1" candidate_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94ca437df08467bae40554a10450269" is_pre_election: true
I20251029 16:34:01.404304 7477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82b72b5cd6a1496eb09ac78395a285a1" candidate_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7cbc2353a042bba0a6103565a2828c"
I20251029 16:34:01.404592 7477 raft_consensus.cc:3060] T 82b72b5cd6a1496eb09ac78395a285a1 P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.406147 7770 raft_consensus.cc:2468] T 82b72b5cd6a1496eb09ac78395a285a1 P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec98b5d57094792a1aff2a1df4e7a46 in term 1.
I20251029 16:34:01.406901 7562 leader_election.cc:304] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269; no voters:
I20251029 16:34:01.410017 7477 raft_consensus.cc:2468] T 82b72b5cd6a1496eb09ac78395a285a1 P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec98b5d57094792a1aff2a1df4e7a46 in term 1.
I20251029 16:34:01.411198 7892 raft_consensus.cc:2804] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 16:34:01.411463 7892 raft_consensus.cc:697] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Running, Role: LEADER
I20251029 16:34:01.411777 7892 consensus_queue.cc:237] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.424628 7888 tablet_bootstrap.cc:492] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269: No bootstrap required, opened a new log
I20251029 16:34:01.424829 7888 ts_tablet_manager.cc:1403] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269: Time spent bootstrapping tablet: real 0.143s user 0.033s sys 0.027s
I20251029 16:34:01.425697 7888 raft_consensus.cc:359] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.425931 7888 raft_consensus.cc:385] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.426012 7888 raft_consensus.cc:740] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ca437df08467bae40554a10450269, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.426256 7888 consensus_queue.cc:260] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.426779 7888 ts_tablet_manager.cc:1434] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 16:34:01.426905 7886 tablet_bootstrap.cc:492] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46: No bootstrap required, opened a new log
I20251029 16:34:01.427085 7886 ts_tablet_manager.cc:1403] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent bootstrapping tablet: real 0.109s user 0.024s sys 0.035s
I20251029 16:34:01.427912 7886 raft_consensus.cc:359] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.428133 7886 raft_consensus.cc:385] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.428241 7886 raft_consensus.cc:740] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.430300 7886 consensus_queue.cc:260] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.434530 7886 ts_tablet_manager.cc:1434] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 16:34:01.438098 7886 tablet_bootstrap.cc:492] T 6a78066ce12c4092b69722de857c7a2a P 7ec98b5d57094792a1aff2a1df4e7a46: Bootstrap starting.
I20251029 16:34:01.440152 7896 raft_consensus.cc:493] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:01.440371 7896 raft_consensus.cc:515] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.441176 7896 leader_election.cc:290] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:01.441344 7886 tablet_bootstrap.cc:654] T 6a78066ce12c4092b69722de857c7a2a P 7ec98b5d57094792a1aff2a1df4e7a46: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.442363 7628 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28e1452396924512ac30d309085628ee" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" is_pre_election: true
I20251029 16:34:01.442693 7628 raft_consensus.cc:2468] T 28e1452396924512ac30d309085628ee P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b94ca437df08467bae40554a10450269 in term 0.
I20251029 16:34:01.443349 7704 leader_election.cc:304] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269; no voters:
I20251029 16:34:01.443812 7896 raft_consensus.cc:2804] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 16:34:01.443971 7896 raft_consensus.cc:493] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 16:34:01.444080 7896 raft_consensus.cc:3060] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.446770 7477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28e1452396924512ac30d309085628ee" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7cbc2353a042bba0a6103565a2828c" is_pre_election: true
W20251029 16:34:01.447824 7706 leader_election.cc:343] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551): Illegal state: must be running to vote when last-logged opid is not known
I20251029 16:34:01.448954 7887 tablet_bootstrap.cc:492] T 28e1452396924512ac30d309085628ee P 4a7cbc2353a042bba0a6103565a2828c: No bootstrap required, opened a new log
I20251029 16:34:01.449110 7887 ts_tablet_manager.cc:1403] T 28e1452396924512ac30d309085628ee P 4a7cbc2353a042bba0a6103565a2828c: Time spent bootstrapping tablet: real 0.190s user 0.040s sys 0.043s
I20251029 16:34:01.449638 7896 raft_consensus.cc:515] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.449994 7887 raft_consensus.cc:359] T 28e1452396924512ac30d309085628ee P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.450239 7887 raft_consensus.cc:385] T 28e1452396924512ac30d309085628ee P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.450341 7887 raft_consensus.cc:740] T 28e1452396924512ac30d309085628ee P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.450640 7887 consensus_queue.cc:260] T 28e1452396924512ac30d309085628ee P 4a7cbc2353a042bba0a6103565a2828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.451212 7887 ts_tablet_manager.cc:1434] T 28e1452396924512ac30d309085628ee P 4a7cbc2353a042bba0a6103565a2828c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 16:34:01.451766 7478 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28e1452396924512ac30d309085628ee" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7cbc2353a042bba0a6103565a2828c"
I20251029 16:34:01.452724 7478 raft_consensus.cc:3060] T 28e1452396924512ac30d309085628ee P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.452889 7887 tablet_bootstrap.cc:492] T 6a78066ce12c4092b69722de857c7a2a P 4a7cbc2353a042bba0a6103565a2828c: Bootstrap starting.
I20251029 16:34:01.454898 7887 tablet_bootstrap.cc:654] T 6a78066ce12c4092b69722de857c7a2a P 4a7cbc2353a042bba0a6103565a2828c: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:01.456450 7478 raft_consensus.cc:2468] T 28e1452396924512ac30d309085628ee P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94ca437df08467bae40554a10450269 in term 1.
I20251029 16:34:01.457608 7886 tablet_bootstrap.cc:492] T 6a78066ce12c4092b69722de857c7a2a P 7ec98b5d57094792a1aff2a1df4e7a46: No bootstrap required, opened a new log
I20251029 16:34:01.457818 7886 ts_tablet_manager.cc:1403] T 6a78066ce12c4092b69722de857c7a2a P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.002s
I20251029 16:34:01.458707 7886 raft_consensus.cc:359] T 6a78066ce12c4092b69722de857c7a2a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.458921 7886 raft_consensus.cc:385] T 6a78066ce12c4092b69722de857c7a2a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.459002 7886 raft_consensus.cc:740] T 6a78066ce12c4092b69722de857c7a2a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.459259 7706 leader_election.cc:304] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c, b94ca437df08467bae40554a10450269; no voters:
I20251029 16:34:01.459297 7886 consensus_queue.cc:260] T 6a78066ce12c4092b69722de857c7a2a P 7ec98b5d57094792a1aff2a1df4e7a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.459872 7954 raft_consensus.cc:2804] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 16:34:01.460079 7954 raft_consensus.cc:697] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [term 1 LEADER]: Becoming Leader. State: Replica: b94ca437df08467bae40554a10450269, State: Running, Role: LEADER
I20251029 16:34:01.460374 7954 consensus_queue.cc:237] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.461892 7886 ts_tablet_manager.cc:1434] T 6a78066ce12c4092b69722de857c7a2a P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 16:34:01.463434 7628 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28e1452396924512ac30d309085628ee" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46"
I20251029 16:34:01.463662 7628 raft_consensus.cc:3060] T 28e1452396924512ac30d309085628ee P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.470135 7628 raft_consensus.cc:2468] T 28e1452396924512ac30d309085628ee P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94ca437df08467bae40554a10450269 in term 1.
I20251029 16:34:01.471046 7896 leader_election.cc:290] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 election: Requested vote from peers 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:01.479775 7887 tablet_bootstrap.cc:492] T 6a78066ce12c4092b69722de857c7a2a P 4a7cbc2353a042bba0a6103565a2828c: No bootstrap required, opened a new log
I20251029 16:34:01.480028 7887 ts_tablet_manager.cc:1403] T 6a78066ce12c4092b69722de857c7a2a P 4a7cbc2353a042bba0a6103565a2828c: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.004s
I20251029 16:34:01.481014 7887 raft_consensus.cc:359] T 6a78066ce12c4092b69722de857c7a2a P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.481263 7887 raft_consensus.cc:385] T 6a78066ce12c4092b69722de857c7a2a P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:01.481335 7887 raft_consensus.cc:740] T 6a78066ce12c4092b69722de857c7a2a P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.481600 7887 consensus_queue.cc:260] T 6a78066ce12c4092b69722de857c7a2a P 4a7cbc2353a042bba0a6103565a2828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.482164 7887 ts_tablet_manager.cc:1434] T 6a78066ce12c4092b69722de857c7a2a P 4a7cbc2353a042bba0a6103565a2828c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20251029 16:34:01.543148 7893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:34:01.543537 7893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:34:01.543658 7893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:34:01.552564 7893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 16:34:01.552688 7893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:34:01.552769 7893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 16:34:01.552835 7893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 16:34:01.563423 7893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.125:44441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.180.125
--webserver_port=41231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:34:01.565450 7893 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:34:01.567473 7893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 16:34:01.577033 7973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:01.577704 7974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:01.578298 7976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:34:01.578856 7893 server_base.cc:1047] running on GCE node
I20251029 16:34:01.579828 7893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 16:34:01.581562 7893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:34:01.582808 7893 hybrid_clock.cc:648] HybridClock initialized: now 1761755641582741 us; error 68 us; skew 500 ppm
I20251029 16:34:01.583205 7893 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:34:01.590265 7893 webserver.cc:492] Webserver started at http://127.6.180.125:41231/ using document root <none> and password file <none>
I20251029 16:34:01.591204 7893 fs_manager.cc:362] Metadata directory not provided
I20251029 16:34:01.591367 7893 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:34:01.598038 7893 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20251029 16:34:01.601589 7982 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:34:01.603406 7893 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20251029 16:34:01.603612 7893 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
uuid: "3155690d4bb54c97b5a1e7881608ae08"
format_stamp: "Formatted at 2025-10-29 16:33:56 on dist-test-slave-l1bh"
I20251029 16:34:01.604508 7893 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:34:01.645292 7954 raft_consensus.cc:493] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:01.645527 7954 raft_consensus.cc:515] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.646430 7954 leader_election.cc:290] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379), 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551)
I20251029 16:34:01.647053 7628 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a78066ce12c4092b69722de857c7a2a" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" is_pre_election: true
I20251029 16:34:01.647300 7628 raft_consensus.cc:2468] T 6a78066ce12c4092b69722de857c7a2a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b94ca437df08467bae40554a10450269 in term 0.
I20251029 16:34:01.647706 7477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a78066ce12c4092b69722de857c7a2a" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7cbc2353a042bba0a6103565a2828c" is_pre_election: true
W20251029 16:34:01.648196 7825 connection.cc:537] client connection to 127.6.180.126:36819 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251029 16:34:01.648273 7704 leader_election.cc:304] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269; no voters:
W20251029 16:34:01.648417 7563 connection.cc:537] client connection to 127.6.180.126:36819 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251029 16:34:01.648800 7893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20251029 16:34:01.649183 7825 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.6.180.126:36819)
I20251029 16:34:01.649181 7954 raft_consensus.cc:2804] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 16:34:01.649300 7954 raft_consensus.cc:493] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 16:34:01.649391 7954 raft_consensus.cc:3060] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Advancing to term 1
W20251029 16:34:01.649824 7825 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.180.126:36819)
I20251029 16:34:01.650012 7893 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:34:01.650331 7893 kserver.cc:163] Server-wide thread pool size limit: 3276
W20251029 16:34:01.650601 7825 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.6.180.126:36819)
W20251029 16:34:01.651506 7825 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.6.180.126:36819)
I20251029 16:34:01.652902 7954 raft_consensus.cc:515] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
W20251029 16:34:01.653736 7929 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 16:34:01.649208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.180.126:36819 (local address 127.6.180.66:54221)
1029 16:34:01.652526 (+ 3318us) negotiation.cc:107] Waiting for socket to connect
1029 16:34:01.652656 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":3168}
W20251029 16:34:01.656221 7952 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 16:34:01.648097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.180.126:36819 (local address 127.6.180.67:53145)
1029 16:34:01.648647 (+ 550us) negotiation.cc:107] Waiting for socket to connect
1029 16:34:01.648669 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1029 16:34:01.648817 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1029 16:34:01.655091 (+ 6274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":432}
I20251029 16:34:01.656936 7477 raft_consensus.cc:2468] T 6a78066ce12c4092b69722de857c7a2a P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b94ca437df08467bae40554a10450269 in term 0.
I20251029 16:34:01.657503 7628 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a78066ce12c4092b69722de857c7a2a" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46"
I20251029 16:34:01.657773 7628 raft_consensus.cc:3060] T 6a78066ce12c4092b69722de857c7a2a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.660784 7477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a78066ce12c4092b69722de857c7a2a" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7cbc2353a042bba0a6103565a2828c"
I20251029 16:34:01.661082 7477 raft_consensus.cc:3060] T 6a78066ce12c4092b69722de857c7a2a P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:01.661711 7628 raft_consensus.cc:2468] T 6a78066ce12c4092b69722de857c7a2a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94ca437df08467bae40554a10450269 in term 1.
I20251029 16:34:01.662416 7704 leader_election.cc:304] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269; no voters:
I20251029 16:34:01.662889 7896 raft_consensus.cc:2804] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 16:34:01.663046 7896 raft_consensus.cc:697] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [term 1 LEADER]: Becoming Leader. State: Replica: b94ca437df08467bae40554a10450269, State: Running, Role: LEADER
I20251029 16:34:01.663478 7896 consensus_queue.cc:237] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:01.665607 7477 raft_consensus.cc:2468] T 6a78066ce12c4092b69722de857c7a2a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94ca437df08467bae40554a10450269 in term 1.
I20251029 16:34:01.666363 7954 leader_election.cc:290] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 election: Requested vote from peers 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379), 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551)
I20251029 16:34:01.691737 7893 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.125:44441
I20251029 16:34:01.691757 8036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.125:44441 every 8 connection(s)
I20251029 16:34:01.693502 7893 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
I20251029 16:34:01.699647 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 7893
I20251029 16:34:01.704365 8037 sys_catalog.cc:263] Verifying existing consensus state
I20251029 16:34:01.711971 8037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Bootstrap starting.
I20251029 16:34:01.716374 8041 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0442da13a653435dba87fc90d8e30d1f)
I20251029 16:34:01.716572 8041 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:01.719344 8041 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
W20251029 16:34:01.721534 7264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:01.724706 7264 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:34:01.729914 8037 log.cc:826] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Log is configured to *not* fsync() on all Append() calls
I20251029 16:34:01.739174 8037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251029 16:34:01.739899 8037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Bootstrap complete.
I20251029 16:34:01.745493 8037 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:01.746309 8037 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3155690d4bb54c97b5a1e7881608ae08, State: Initialized, Role: FOLLOWER
I20251029 16:34:01.747025 8037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:01.748706 8046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:01.749034 8046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:01.749974 8037 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: configured and running, proceeding with master startup.
I20251029 16:34:01.750236 8012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "3155690d4bb54c97b5a1e7881608ae08" is_pre_election: true
I20251029 16:34:01.750607 7922 consensus_queue.cc:1048] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:01.751080 8012 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f9bab9976745478595681935778cac in term 2.
I20251029 16:34:01.751749 7266 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f
I20251029 16:34:01.752102 8041 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251029 16:34:01.752240 8041 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0442da13a653435dba87fc90d8e30d1f)
I20251029 16:34:01.752346 8041 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 3 FOLLOWER]: Advancing to term 4
I20251029 16:34:01.758937 8041 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:01.759963 8041 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 4 election: Requested vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
I20251029 16:34:01.761373 8012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "3155690d4bb54c97b5a1e7881608ae08"
I20251029 16:34:01.761649 8012 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 2 FOLLOWER]: Advancing to term 4
W20251029 16:34:01.761790 7264 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:34:01.768245 8012 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f9bab9976745478595681935778cac in term 4.
I20251029 16:34:01.768385 7819 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:34:01.768618 7819 heartbeater.cc:499] Master 127.6.180.125:44441 was elected leader, sending a full tablet report...
I20251029 16:34:01.772928 7819 heartbeater.cc:499] Master 127.6.180.125:44441 was elected leader, sending a full tablet report...
I20251029 16:34:01.774186 7920 consensus_queue.cc:1048] T 1ddd091e76844d0cabcd2ea2ffe767cc P 4a7cbc2353a042bba0a6103565a2828c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:01.783185 7266 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f
I20251029 16:34:01.783746 8041 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 4 FOLLOWER]: Leader election won for term 4
I20251029 16:34:01.793361 7526 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:34:01.796330 7677 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:34:01.800276 8041 raft_consensus.cc:697] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 4 LEADER]: Becoming Leader. State: Replica: 29f9bab9976745478595681935778cac, State: Running, Role: LEADER
I20251029 16:34:01.800854 8041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 3.34, Last appended by leader: 34, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:01.804586 8058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f9bab9976745478595681935778cac. Latest consensus state: current_term: 4 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:01.805034 8058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: LEADER
I20251029 16:34:01.807365 8068 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251029 16:34:01.811013 8068 catalog_manager.cc:679] Loaded metadata for table default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe]
I20251029 16:34:01.811753 8068 catalog_manager.cc:679] Loaded metadata for table default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab]
I20251029 16:34:01.812467 8068 catalog_manager.cc:679] Loaded metadata for table default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c]
I20251029 16:34:01.813128 8068 catalog_manager.cc:679] Loaded metadata for table default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0]
I20251029 16:34:01.813717 8068 catalog_manager.cc:679] Loaded metadata for table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]
I20251029 16:34:01.814375 8068 catalog_manager.cc:679] Loaded metadata for table default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b]
I20251029 16:34:01.814975 8068 catalog_manager.cc:679] Loaded metadata for table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]
I20251029 16:34:01.815586 8068 catalog_manager.cc:679] Loaded metadata for table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]
I20251029 16:34:01.816206 8068 catalog_manager.cc:679] Loaded metadata for table default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6]
I20251029 16:34:01.816967 8069 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:01.817096 8069 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: loading cluster ID for follower catalog manager: success
I20251029 16:34:01.818634 8068 tablet_loader.cc:96] loaded metadata for tablet 1ddd091e76844d0cabcd2ea2ffe767cc (table default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6])
I20251029 16:34:01.819135 8068 tablet_loader.cc:96] loaded metadata for tablet 28e1452396924512ac30d309085628ee (table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f])
I20251029 16:34:01.819579 8068 tablet_loader.cc:96] loaded metadata for tablet 41ad6aef032040f2933152a36de8d120 (table default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c])
I20251029 16:34:01.820046 8068 tablet_loader.cc:96] loaded metadata for tablet 4233e0df604c4ff8a7e60528788d7d07 (table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80])
I20251029 16:34:01.820536 8068 tablet_loader.cc:96] loaded metadata for tablet 6a78066ce12c4092b69722de857c7a2a (table default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe])
I20251029 16:34:01.821033 8068 tablet_loader.cc:96] loaded metadata for tablet 82b72b5cd6a1496eb09ac78395a285a1 (table default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0])
I20251029 16:34:01.822134 8068 tablet_loader.cc:96] loaded metadata for tablet a6d6b65d23ce4b84aab753d124d9e1cf (table default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab])
I20251029 16:34:01.822902 8068 tablet_loader.cc:96] loaded metadata for tablet af86f1498654491da54d0da20774edd5 (table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e])
I20251029 16:34:01.823686 8068 tablet_loader.cc:96] loaded metadata for tablet e1ac79eb37644b3e8e02681941c20091 (table default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b])
I20251029 16:34:01.824127 8068 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251029 16:34:01.826006 8068 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:01.826223 8068 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251029 16:34:01.828876 8068 catalog_manager.cc:1514] Loading token signing keys...
I20251029 16:34:01.830307 8068 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Loaded TSK: 0
I20251029 16:34:01.831576 8068 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251029 16:34:01.832947 8068 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251029 16:34:01.834291 8068 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251029 16:34:01.834991 8069 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: acquiring CA information for follower catalog manager: success
I20251029 16:34:01.829895 7893 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:34:01.836068 7893 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:01.838306 8069 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 16:34:01.840058 8054 consensus_queue.cc:1048] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:01.844007 7279 master_service.cc:954] ReplaceTablet: received request to replace tablet 4233e0df604c4ff8a7e60528788d7d07 from {username='slave'} at 127.0.0.1:37764
I20251029 16:34:01.844558 7276 master_service.cc:954] ReplaceTablet: received request to replace tablet 4233e0df604c4ff8a7e60528788d7d07 from {username='slave'} at 127.0.0.1:37764
I20251029 16:34:01.845919 7277 master_service.cc:954] ReplaceTablet: received request to replace tablet 1ddd091e76844d0cabcd2ea2ffe767cc from {username='slave'} at 127.0.0.1:37764
I20251029 16:34:01.846792 7677 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:34:01.847800 7278 master_service.cc:954] ReplaceTablet: received request to replace tablet 82b72b5cd6a1496eb09ac78395a285a1 from {username='slave'} at 127.0.0.1:37764
I20251029 16:34:01.852262 7272 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37764:
name: "default.Table_cf493c1cc4f44b069dba62ecf1ad3676"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:01.854137 7274 master_service.cc:954] ReplaceTablet: received request to replace tablet 6a78066ce12c4092b69722de857c7a2a from {username='slave'} at 127.0.0.1:37764
I20251029 16:34:01.855371 7270 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37764:
table { table_name: "default.Table_0982a1f0341d4f3f9e7f27ed71278b4d" } modify_external_catalogs: true
I20251029 16:34:01.856837 7270 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37764:
table { table_name: "default.Table_0982a1f0341d4f3f9e7f27ed71278b4d" } modify_external_catalogs: true
I20251029 16:34:01.850332 7273 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37764:
name: "default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:01.861068 8001 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" instance_seqno: 1761755639071323) as {username='slave'} at 127.6.180.66:48173; Asking this server to re-register.
I20251029 16:34:01.855993 7275 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37764:
name: "default.Table_75040a14be804518a951b603c808d402"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:01.863530 7920 consensus_queue.cc:1048] T af86f1498654491da54d0da20774edd5 P 4a7cbc2353a042bba0a6103565a2828c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:01.862905 7271 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37764:
name: "default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:01.866039 7819 heartbeater.cc:499] Master 127.6.180.125:44441 was elected leader, sending a full tablet report...
I20251029 16:34:01.867854 7677 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:01.869743 8001 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" instance_seqno: 1761755638480543) as {username='slave'} at 127.6.180.65:52167; Asking this server to re-register.
I20251029 16:34:01.871001 7526 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:01.871860 8012 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Refusing update from remote peer 29f9bab9976745478595681935778cac: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
W20251029 16:34:01.872057 7264 consensus_peers.cc:597] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac -> Peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Couldn't send request to peer 0442da13a653435dba87fc90d8e30d1f. Status: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251029 16:34:01.873190 8041 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 5, Time since last communication: 0.000s
I20251029 16:34:01.874091 8001 ts_manager.cc:194] Registered new tserver with Master: 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:01.874279 8002 ts_manager.cc:194] Registered new tserver with Master: 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551)
F20251029 16:34:01.884429 7354 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1761755639, eventType=CREATE_TABLE, dbName=default, tableName=table_b513b87e8a1943eea3e24140ede4921d, message={"server":"","servicePrincipal":"","db":"default","table":"table_b513b87e8a1943eea3e24140ede4921d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b513b87e8a1943eea3e24140ede4921d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761755639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/warehouse/table_b513b87e8a1943eea3e24140ede4921d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b513b87e8a1943eea3e24140ede4921d\",\"kudu.table_id\":\"1398ac3c2f22427b96af2e9ff15e62d4\",\"transient_lastDdlTime\":\"1761755639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"415c0df9d5524c9a82ceee0f02442bac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761755640,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761755640, eventType=CREATE_TABLE, dbName=default, tableName=table_4d4ea90e32c7496a91ee99d2ca5a1f01, message={"server":"","servicePrincipal":"","db":"default","table":"table_4d4ea90e32c7496a91ee99d2ca5a1f01","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_4d4ea90e32c7496a91ee99d2ca5a1f01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761755640},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/warehouse/table_4d4ea90e32c7496a91ee99d2ca5a1f01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_4d4ea90e32c7496a91ee99d2ca5a1f01\",\"kudu.table_id\":\"d31229edb9834ddab88752bc8d367e80\",\"transient_lastDdlTime\":\"1761755640\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"415c0df9d5524c9a82ceee0f02442bac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761755640,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f1c27162dcd google::LogMessage::Fail() at ??:0
@ 0x7f1c27166b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f1c271627cc google::LogMessage::Flush() at ??:0
@ 0x7f1c27162a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f1c3db24b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f1c3db23411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f1c3db2b08c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251029 16:34:01.912897 8046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f9bab9976745478595681935778cac. Latest consensus state: current_term: 4 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:01.913197 8046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:01.914368 8002 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b94ca437df08467bae40554a10450269" instance_seqno: 1761755639575696) as {username='slave'} at 127.6.180.67:34007; Asking this server to re-register.
I20251029 16:34:01.914834 8080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:01.915182 8080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:01.917774 7819 heartbeater.cc:461] Registering TS with master...
@ 0x7f1c3db2af09 std::_Function_handler<>::_M_invoke() at ??:0
I20251029 16:34:01.920293 8080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:01.920496 8080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:01.922459 8002 ts_manager.cc:194] Registered new tserver with Master: b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
@ 0xa8867d std::function<>::operator()() at ??:?
I20251029 16:34:01.928108 7279 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Sending DeleteTablet for 3 replicas of tablet 4233e0df604c4ff8a7e60528788d7d07
@ 0x7f1c29ed6ee9 kudu::Thread::SuperviseThread() at ??:0
I20251029 16:34:01.930557 7279 catalog_manager.cc:6719] ReplaceTablet: tablet 4233e0df604c4ff8a7e60528788d7d07 deleted and replaced by tablet 167c0aa33db046408bbc9bcd0844d152
@ 0x7f1c2d4e16db start_thread at ??:0
@ 0x7f1c2601371f clone at ??:0
I20251029 16:34:01.963294 8076 mvcc.cc:204] Tried to move back new op lower bound from 7216151109089583104 to 7216151101362245632. Current Snapshot: MvccSnapshot[applied={T|T < 7216151109089583104 or (T in {7216151109089583104})}]
I20251029 16:34:02.026410 7892 consensus_queue.cc:1048] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:02.037215 7892 consensus_queue.cc:1048] T e1ac79eb37644b3e8e02681941c20091 P 7ec98b5d57094792a1aff2a1df4e7a46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:02.163342 7920 consensus_queue.cc:1048] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 16:34:02.175595 8091 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 16:34:01.932012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.6.180.67:38629)
1029 16:34:01.937530 (+ 5518us) server_negotiation.cc:207] Beginning negotiation
1029 16:34:01.937537 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1029 16:34:02.175270 (+237733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49932: BlockingRecv error: recv got EOF from 127.0.0.1:49932 (error 108)
Metrics: {"server-negotiator.queue_time_us":5398,"thread_start_us":5290,"threads_started":1}
W20251029 16:34:02.175717 8090 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 16:34:01.931585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.6.180.66:33379)
1029 16:34:01.936486 (+ 4901us) server_negotiation.cc:207] Beginning negotiation
1029 16:34:01.936496 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1029 16:34:01.936539 (+ 43us) server_negotiation.cc:408] Connection header received
1029 16:34:01.936658 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1029 16:34:01.936670 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
1029 16:34:01.936789 (+ 119us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1029 16:34:01.936963 (+ 174us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1029 16:34:02.175392 (+238429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35044: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":4708,"thread_start_us":4607,"threads_started":1}
W20251029 16:34:02.176213 7825 connection.cc:537] client connection to 127.6.180.124:39271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251029 16:34:02.180866 7922 consensus_queue.cc:1048] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 16:34:02.186419 8088 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 16:34:01.931245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55880 (local address 127.6.180.65:39551)
1029 16:34:01.938015 (+ 6770us) server_negotiation.cc:207] Beginning negotiation
1029 16:34:01.938024 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1029 16:34:01.938047 (+ 23us) server_negotiation.cc:408] Connection header received
1029 16:34:01.938125 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1029 16:34:01.938130 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1029 16:34:01.938240 (+ 110us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1029 16:34:01.938357 (+ 117us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1029 16:34:02.185354 (+246997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55880: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":6641,"thread_start_us":6523,"threads_started":1}
W20251029 16:34:02.187372 7989 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 16:34:02.181782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.180.124:39271 (local address 127.0.0.1:37778)
1029 16:34:02.181930 (+ 148us) negotiation.cc:107] Waiting for socket to connect
1029 16:34:02.181945 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1029 16:34:02.182059 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1029 16:34:02.187077 (+ 5018us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":41}
W20251029 16:34:02.188223 7917 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 16:34:02.184462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.180.124:39271 (local address 127.6.180.65:41187)
1029 16:34:02.184705 (+ 243us) negotiation.cc:107] Waiting for socket to connect
1029 16:34:02.184724 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1029 16:34:02.184839 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1029 16:34:02.188000 (+ 3161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":143}
W20251029 16:34:02.189067 8057 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 16:34:02.184791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.180.124:39271 (local address 127.6.180.66:34977)
1029 16:34:02.185101 (+ 310us) negotiation.cc:107] Waiting for socket to connect
1029 16:34:02.185117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1029 16:34:02.185242 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1029 16:34:02.188793 (+ 3551us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":193}
W20251029 16:34:02.198699 7952 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 16:34:02.186327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.180.124:39271 (local address 127.6.180.67:36431)
1029 16:34:02.186765 (+ 438us) negotiation.cc:107] Waiting for socket to connect
1029 16:34:02.186787 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1029 16:34:02.186921 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1029 16:34:02.198492 (+ 11571us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":302}
I20251029 16:34:02.239554 8100 raft_consensus.cc:493] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:02.239745 8100 raft_consensus.cc:515] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } }
I20251029 16:34:02.240499 8100 leader_election.cc:290] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551), b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:02.241070 7770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4233e0df604c4ff8a7e60528788d7d07" candidate_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94ca437df08467bae40554a10450269" is_pre_election: true
I20251029 16:34:02.241134 7477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4233e0df604c4ff8a7e60528788d7d07" candidate_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7cbc2353a042bba0a6103565a2828c" is_pre_election: true
I20251029 16:34:02.241396 7770 raft_consensus.cc:2393] T 4233e0df604c4ff8a7e60528788d7d07 P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ec98b5d57094792a1aff2a1df4e7a46 in current term 1: Already voted for candidate 4a7cbc2353a042bba0a6103565a2828c in this term.
I20251029 16:34:02.241955 7564 leader_election.cc:304] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ec98b5d57094792a1aff2a1df4e7a46; no voters: 4a7cbc2353a042bba0a6103565a2828c, b94ca437df08467bae40554a10450269
I20251029 16:34:02.242254 8100 raft_consensus.cc:3060] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:02.244976 8100 raft_consensus.cc:2749] T 4233e0df604c4ff8a7e60528788d7d07 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 16:34:02.258572 8100 consensus_queue.cc:1048] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:02.265549 8109 consensus_queue.cc:1048] T 82b72b5cd6a1496eb09ac78395a285a1 P 7ec98b5d57094792a1aff2a1df4e7a46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251029 16:34:02.274693 7920 consensus_queue.cc:1048] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:02.277451 7896 consensus_queue.cc:1048] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:02.293341 8054 consensus_queue.cc:1048] T 4233e0df604c4ff8a7e60528788d7d07 P 4a7cbc2353a042bba0a6103565a2828c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:02.302309 8114 consensus_queue.cc:1048] T 28e1452396924512ac30d309085628ee P b94ca437df08467bae40554a10450269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251029 16:34:02.333135 8117 consensus_queue.cc:1048] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:02.338913 8115 consensus_queue.cc:1048] T a6d6b65d23ce4b84aab753d124d9e1cf P b94ca437df08467bae40554a10450269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:02.422659 8121 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 29f9bab9976745478595681935778cac)
I20251029 16:34:02.422847 8121 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:02.424780 8121 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:02.425745 7984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:02.427469 7984 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:02.427706 7984 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:02.427810 7984 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:02.428195 8121 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251029 16:34:02.437866 8114 consensus_queue.cc:1048] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:02.445871 8115 consensus_queue.cc:1048] T 6a78066ce12c4092b69722de857c7a2a P b94ca437df08467bae40554a10450269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:03.174134 8131 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 29f9bab9976745478595681935778cac)
I20251029 16:34:03.174335 8131 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:03.175069 8131 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:03.177553 7984 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:03.177753 7984 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:03.177862 7984 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:03.178184 8131 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251029 16:34:04.227255 8148 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 29f9bab9976745478595681935778cac)
I20251029 16:34:04.227516 8148 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:04.228447 8148 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:04.231056 7984 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:04.231284 7984 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:04.231391 7984 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:04.231673 8148 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251029 16:34:04.651851 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.124:39271
--webserver_interface=127.6.180.124
--webserver_port=46685
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251029 16:34:05.018324 8156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:34:05.018684 8156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:34:05.018766 8156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:34:05.027238 8156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 16:34:05.027345 8156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:34:05.027408 8156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 16:34:05.027477 8156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 16:34:05.038877 8156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.124:39271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.180.124
--webserver_port=46685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:34:05.040771 8156 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:34:05.042744 8156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 16:34:05.052290 8163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:05.052597 8166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:05.051806 8164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:34:05.053081 8156 server_base.cc:1047] running on GCE node
I20251029 16:34:05.053947 8156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 16:34:05.055644 8156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:34:05.056880 8156 hybrid_clock.cc:648] HybridClock initialized: now 1761755645056806 us; error 70 us; skew 500 ppm
I20251029 16:34:05.057302 8156 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:34:05.059988 8156 webserver.cc:492] Webserver started at http://127.6.180.124:46685/ using document root <none> and password file <none>
I20251029 16:34:05.060892 8156 fs_manager.cc:362] Metadata directory not provided
I20251029 16:34:05.061038 8156 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:34:05.068611 8156 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20251029 16:34:05.072064 8173 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:34:05.073731 8156 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20251029 16:34:05.073951 8156 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
uuid: "29f9bab9976745478595681935778cac"
format_stamp: "Formatted at 2025-10-29 16:33:57 on dist-test-slave-l1bh"
I20251029 16:34:05.074792 8156 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:34:05.110355 8156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 16:34:05.111317 8156 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:34:05.111613 8156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 16:34:05.133204 8156 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.124:39271
I20251029 16:34:05.133277 8224 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.124:39271 every 8 connection(s)
I20251029 16:34:05.135083 8156 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
I20251029 16:34:05.143952 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 8156
I20251029 16:34:05.148272 8225 sys_catalog.cc:263] Verifying existing consensus state
I20251029 16:34:05.156579 8225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Bootstrap starting.
I20251029 16:34:05.171348 8225 log.cc:826] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Log is configured to *not* fsync() on all Append() calls
I20251029 16:34:05.176961 7675 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:34:05.204619 8225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Bootstrap replayed 1/1 log segments. Stats: ops{read=39 overwritten=0 applied=36 ignored=0} inserts{seen=26 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20251029 16:34:05.205353 8225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Bootstrap complete.
I20251029 16:34:05.211150 8225 raft_consensus.cc:359] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 4 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:05.214686 8225 raft_consensus.cc:740] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f9bab9976745478595681935778cac, State: Initialized, Role: FOLLOWER
I20251029 16:34:05.215533 8225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 4.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:05.216431 8235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:05.216799 8235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:05.217924 8225 sys_catalog.cc:565] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: configured and running, proceeding with master startup.
I20251029 16:34:05.231518 8249 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:05.231807 8249 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: loading cluster ID for follower catalog manager: success
I20251029 16:34:05.243217 8249 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: acquiring CA information for follower catalog manager: success
I20251029 16:34:05.246881 8249 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 16:34:05.242362 8156 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:34:05.249110 8156 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:05.419363 7817 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:34:05.422583 8190 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b94ca437df08467bae40554a10450269" instance_seqno: 1761755639575696) as {username='slave'} at 127.6.180.67:58397; Asking this server to re-register.
I20251029 16:34:05.423432 7817 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:05.425485 8190 ts_manager.cc:194] Registered new tserver with Master: b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:05.429296 8254 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 29f9bab9976745478595681935778cac)
I20251029 16:34:05.429502 8254 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:05.430190 8254 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:05.433588 7984 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:34:05.439803 8200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3155690d4bb54c97b5a1e7881608ae08" candidate_term: 5 candidate_status { last_received { term: 4 index: 39 } } ignore_live_leader: false dest_uuid: "29f9bab9976745478595681935778cac" is_pre_election: true
I20251029 16:34:05.440482 8200 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3155690d4bb54c97b5a1e7881608ae08 in term 4.
I20251029 16:34:05.441267 7984 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f
I20251029 16:34:05.441608 8254 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251029 16:34:05.441745 8254 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 29f9bab9976745478595681935778cac)
I20251029 16:34:05.441864 8254 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 4 FOLLOWER]: Advancing to term 5
I20251029 16:34:05.444132 8254 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:05.444824 8254 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 election: Requested vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
I20251029 16:34:05.445712 8200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3155690d4bb54c97b5a1e7881608ae08" candidate_term: 5 candidate_status { last_received { term: 4 index: 39 } } ignore_live_leader: false dest_uuid: "29f9bab9976745478595681935778cac"
I20251029 16:34:05.445927 8200 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 4 FOLLOWER]: Advancing to term 5
W20251029 16:34:05.448819 7984 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:34:05.450445 8200 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3155690d4bb54c97b5a1e7881608ae08 in term 5.
I20251029 16:34:05.450941 7984 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f
I20251029 16:34:05.451345 8254 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 5 FOLLOWER]: Leader election won for term 5
I20251029 16:34:05.451805 8254 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 5 LEADER]: Becoming Leader. State: Replica: 3155690d4bb54c97b5a1e7881608ae08, State: Running, Role: LEADER
I20251029 16:34:05.452311 8254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 4.39, Last appended by leader: 39, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:05.454027 8258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3155690d4bb54c97b5a1e7881608ae08. Latest consensus state: current_term: 5 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:05.454330 8258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: LEADER
I20251029 16:34:05.455044 8260 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251029 16:34:05.457654 8260 catalog_manager.cc:679] Loaded metadata for table default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe]
I20251029 16:34:05.458128 8260 catalog_manager.cc:679] Loaded metadata for table default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab]
I20251029 16:34:05.458490 8260 catalog_manager.cc:679] Loaded metadata for table default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c]
I20251029 16:34:05.458866 8260 catalog_manager.cc:679] Loaded metadata for table default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0]
I20251029 16:34:05.459197 8260 catalog_manager.cc:679] Loaded metadata for table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]
I20251029 16:34:05.459543 8260 catalog_manager.cc:679] Loaded metadata for table default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b]
I20251029 16:34:05.459853 8260 catalog_manager.cc:679] Loaded metadata for table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]
I20251029 16:34:05.460270 8260 catalog_manager.cc:679] Loaded metadata for table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]
I20251029 16:34:05.460599 8260 catalog_manager.cc:679] Loaded metadata for table default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6]
I20251029 16:34:05.462292 8260 tablet_loader.cc:96] loaded metadata for tablet 167c0aa33db046408bbc9bcd0844d152 (table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80])
I20251029 16:34:05.462854 8260 tablet_loader.cc:96] loaded metadata for tablet 28e1452396924512ac30d309085628ee (table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f])
I20251029 16:34:05.463169 8260 tablet_loader.cc:96] loaded metadata for tablet 41ad6aef032040f2933152a36de8d120 (table default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c])
I20251029 16:34:05.463573 8260 tablet_loader.cc:96] loaded metadata for tablet 4af807ac4f78451592a0a6788af12044 (table default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6])
I20251029 16:34:05.464044 8260 tablet_loader.cc:96] loaded metadata for tablet 8197c292e24348e1b633ba29e22a0bec (table default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0])
I20251029 16:34:05.464977 8260 tablet_loader.cc:96] loaded metadata for tablet a6d6b65d23ce4b84aab753d124d9e1cf (table default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab])
I20251029 16:34:05.465570 8260 tablet_loader.cc:96] loaded metadata for tablet acbbc2208b744b6c8ce5ea0fe699d29e (table default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe])
I20251029 16:34:05.465902 8260 tablet_loader.cc:96] loaded metadata for tablet af86f1498654491da54d0da20774edd5 (table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e])
I20251029 16:34:05.466517 8260 tablet_loader.cc:96] loaded metadata for tablet e1ac79eb37644b3e8e02681941c20091 (table default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b])
I20251029 16:34:05.466809 8260 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251029 16:34:05.467975 8260 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:05.468066 8260 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251029 16:34:05.469374 8260 catalog_manager.cc:1514] Loading token signing keys...
I20251029 16:34:05.470300 8260 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Loaded TSK: 0
I20251029 16:34:05.471236 8260 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251029 16:34:05.472227 8260 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251029 16:34:05.473069 8260 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251029 16:34:05.488730 8002 master_service.cc:954] ReplaceTablet: received request to replace tablet e1ac79eb37644b3e8e02681941c20091 from {username='slave'} at 127.0.0.1:40970
I20251029 16:34:05.489729 7998 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40970:
table { table_name: "default.Table_4d4ea90e32c7496a91ee99d2ca5a1f01" } new_table_name: "default.Table_2ce7bbbb663d4ced9e2a662c79543a9e" modify_external_catalogs: true
I20251029 16:34:05.490022 7999 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40970:
table { table_name: "default.Table_571d6fa0fe7e4782b8b0c692012fbd81" } new_table_name: "default.Table_ca38a68ba7574beab9c638e82db00946" modify_external_catalogs: true
I20251029 16:34:05.490527 8000 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40970:
table { table_name: "default.Table_809b2f55b3594aa1aab4872f17b07c4e" } new_table_name: "default.Table_feb27132a62f47388739326e4fb4118c" modify_external_catalogs: true
I20251029 16:34:05.490842 7997 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40970:
name: "default.Table_4bedb57357b24f78a75d1f362473ceff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:05.489809 7996 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40970:
table { table_name: "default.Table_dfc244c581534b678f83267a7d10b58c" } new_table_name: "default.Table_35daf50c962249fd983f0e49e1f7960a" modify_external_catalogs: true
I20251029 16:34:05.489756 8001 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40970:
table { table_name: "default.Table_6129e3888a3346669ab7c8982593f438" } new_table_name: "default.Table_323e8fdbeff34a12add0ad9e4c97accd" modify_external_catalogs: true
I20251029 16:34:05.502815 7995 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40970:
name: "default.Table_3a47f0a3057e46acaf74a5e33bffb30a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:05.502977 7994 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40970:
name: "default.Table_e4dfb9e8508e41669fcd08f8c79b24f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:05.503571 7993 master_service.cc:954] ReplaceTablet: received request to replace tablet 6a78066ce12c4092b69722de857c7a2a from {username='slave'} at 127.0.0.1:40970
I20251029 16:34:05.504660 7993 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40970:
name: "default.Table_75040a14be804518a951b603c808d402"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251029 16:34:05.510762 8067 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1761755639, eventType=CREATE_TABLE, dbName=default, tableName=table_b513b87e8a1943eea3e24140ede4921d, message={"server":"","servicePrincipal":"","db":"default","table":"table_b513b87e8a1943eea3e24140ede4921d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b513b87e8a1943eea3e24140ede4921d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761755639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/warehouse/table_b513b87e8a1943eea3e24140ede4921d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b513b87e8a1943eea3e24140ede4921d\",\"kudu.table_id\":\"1398ac3c2f22427b96af2e9ff15e62d4\",\"transient_lastDdlTime\":\"1761755639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"415c0df9d5524c9a82ceee0f02442bac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761755640,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761755640, eventType=CREATE_TABLE, dbName=default, tableName=table_4d4ea90e32c7496a91ee99d2ca5a1f01, message={"server":"","servicePrincipal":"","db":"default","table":"table_4d4ea90e32c7496a91ee99d2ca5a1f01","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_4d4ea90e32c7496a91ee99d2ca5a1f01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761755640},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/warehouse/table_4d4ea90e32c7496a91ee99d2ca5a1f01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_4d4ea90e32c7496a91ee99d2ca5a1f01\",\"kudu.table_id\":\"d31229edb9834ddab88752bc8d367e80\",\"transient_lastDdlTime\":\"1761755640\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"415c0df9d5524c9a82ceee0f02442bac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761755640,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fe567aa7dcd google::LogMessage::Fail() at ??:0
W20251029 16:34:05.514250 7984 consensus_peers.cc:597] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 -> Peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Couldn't send request to peer 0442da13a653435dba87fc90d8e30d1f. Status: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251029 16:34:05.514871 8200 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Refusing update from remote peer 3155690d4bb54c97b5a1e7881608ae08: Log matching property violated. Preceding OpId in replica: term: 4 index: 39. Preceding OpId from leader: term: 5 index: 41. (index mismatch)
@ 0x7fe567aabb93 google::LogMessage::SendToLog() at ??:0
I20251029 16:34:05.515866 8254 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 36, Time since last communication: 0.000s
@ 0x7fe567aa77cc google::LogMessage::Flush() at ??:0
@ 0x7fe567aa7a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fe57e469b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251029 16:34:05.536767 8235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3155690d4bb54c97b5a1e7881608ae08. Latest consensus state: current_term: 5 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:05.537109 8235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fe57e468411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251029 16:34:05.543167 8234 mvcc.cc:204] Tried to move back new op lower bound from 7216151124009340928 to 7216151123776225280. Current Snapshot: MvccSnapshot[applied={T|T < 7216151124009340928}]
I20251029 16:34:05.543545 8235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:05.543756 8235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fe57e47008c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251029 16:34:05.548135 8258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:05.548441 8258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: LEADER
I20251029 16:34:05.549558 8002 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Sending DeleteTablet for 3 replicas of tablet e1ac79eb37644b3e8e02681941c20091
@ 0x7fe57e46ff09 std::_Function_handler<>::_M_invoke() at ??:0
I20251029 16:34:05.551843 8002 catalog_manager.cc:6719] ReplaceTablet: tablet e1ac79eb37644b3e8e02681941c20091 deleted and replaced by tablet f5e3fd53975145e9acbc9dff7a2b0cb9
@ 0xa8867d std::function<>::operator()() at ??:?
I20251029 16:34:05.559031 8002 master_service.cc:954] ReplaceTablet: received request to replace tablet 1ddd091e76844d0cabcd2ea2ffe767cc from {username='slave'} at 127.0.0.1:40970
I20251029 16:34:05.563860 8002 master_service.cc:954] ReplaceTablet: received request to replace tablet 4233e0df604c4ff8a7e60528788d7d07 from {username='slave'} at 127.0.0.1:40970
I20251029 16:34:05.565891 8002 master_service.cc:954] ReplaceTablet: received request to replace tablet 82b72b5cd6a1496eb09ac78395a285a1 from {username='slave'} at 127.0.0.1:40970
@ 0x7fe56a81bee9 kudu::Thread::SuperviseThread() at ??:0
I20251029 16:34:05.566727 8002 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40970:
name: "default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fe56de266db start_thread at ??:0
@ 0x7fe56695871f clone at ??:0
I20251029 16:34:05.612900 7524 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:34:05.621171 8190 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" instance_seqno: 1761755638480543) as {username='slave'} at 127.6.180.65:37603; Asking this server to re-register.
I20251029 16:34:05.622416 7524 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:05.624419 8190 ts_manager.cc:194] Registered new tserver with Master: 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551)
I20251029 16:34:05.683521 8190 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" instance_seqno: 1761755639071323) as {username='slave'} at 127.6.180.66:52791; Asking this server to re-register.
I20251029 16:34:05.684752 7675 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:05.686812 8190 ts_manager.cc:194] Registered new tserver with Master: 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:34:05.824298 8272 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 16:34:05.552855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.6.180.66:33379)
1029 16:34:05.565045 (+ 12190us) server_negotiation.cc:207] Beginning negotiation
1029 16:34:05.565053 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1029 16:34:05.565075 (+ 22us) server_negotiation.cc:408] Connection header received
1029 16:34:05.565172 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1029 16:34:05.565178 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1029 16:34:05.565276 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1029 16:34:05.572781 (+ 7505us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1029 16:34:05.823963 (+251182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49056: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":12064,"thread_start_us":12514,"threads_started":1}
W20251029 16:34:05.824338 8274 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 16:34:05.558826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.6.180.67:38629)
1029 16:34:05.604764 (+ 45938us) server_negotiation.cc:207] Beginning negotiation
1029 16:34:05.604772 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1029 16:34:05.604799 (+ 27us) server_negotiation.cc:408] Connection header received
1029 16:34:05.604891 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1029 16:34:05.604897 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1029 16:34:05.604993 (+ 96us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1029 16:34:05.605118 (+ 125us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1029 16:34:05.823969 (+218851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55708: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":45759,"thread_start_us":46515,"threads_started":1}
W20251029 16:34:05.824867 7565 connection.cc:537] client connection to 127.6.180.125:44441 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:05.824966 7827 connection.cc:537] client connection to 127.6.180.125:44441 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:05.825526 8270 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 16:34:05.552340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.6.180.65:39551)
1029 16:34:05.565110 (+ 12770us) server_negotiation.cc:207] Beginning negotiation
1029 16:34:05.565117 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1029 16:34:05.565136 (+ 19us) server_negotiation.cc:408] Connection header received
1029 16:34:05.565206 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1029 16:34:05.565211 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1029 16:34:05.565286 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1029 16:34:05.565433 (+ 147us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1029 16:34:05.573124 (+ 7691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1029 16:34:05.574706 (+ 1582us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1029 16:34:05.825248 (+250542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54450: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":12639,"thread_start_us":14681,"threads_started":1}
W20251029 16:34:05.825722 7827 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.6.180.125:44441)
W20251029 16:34:05.826316 7827 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.180.125:44441)
W20251029 16:34:05.826779 7827 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.6.180.125:44441)
W20251029 16:34:05.827308 7827 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.6.180.125:44441)
I20251029 16:34:06.068688 8285 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3155690d4bb54c97b5a1e7881608ae08)
I20251029 16:34:06.068953 8285 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:06.070911 8285 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
W20251029 16:34:06.072197 8177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
W20251029 16:34:06.073381 8175 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:06.073565 8177 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:06.073729 8177 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f9bab9976745478595681935778cac; no voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:06.074070 8285 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251029 16:34:06.583206 8291 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3155690d4bb54c97b5a1e7881608ae08)
I20251029 16:34:06.583442 8291 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:06.584156 8291 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
W20251029 16:34:06.590273 8175 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:06.590451 8177 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:06.590574 8177 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f9bab9976745478595681935778cac; no voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:06.590888 8291 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251029 16:34:07.204249 8296 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3155690d4bb54c97b5a1e7881608ae08)
I20251029 16:34:07.204488 8296 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:07.205144 8296 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
W20251029 16:34:07.207404 8175 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:07.207597 8177 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:07.207752 8177 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f9bab9976745478595681935778cac; no voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:07.208089 8296 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251029 16:34:08.313704 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.126:36819
--webserver_interface=127.6.180.126
--webserver_port=40759
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251029 16:34:08.450660 8303 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3155690d4bb54c97b5a1e7881608ae08)
I20251029 16:34:08.450891 8303 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:08.451663 8303 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
W20251029 16:34:08.453863 8175 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:08.454406 8177 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:08.454573 8177 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f9bab9976745478595681935778cac; no voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:08.454841 8303 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251029 16:34:08.681885 8300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:34:08.682262 8300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:34:08.682375 8300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:34:08.691241 8300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 16:34:08.691378 8300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:34:08.691466 8300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 16:34:08.691535 8300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 16:34:08.702420 8300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.126:36819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.180.126
--webserver_port=40759
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:34:08.704366 8300 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:34:08.706708 8300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 16:34:08.715822 8311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:08.715814 8312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:08.716133 8314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:34:08.716804 8300 server_base.cc:1047] running on GCE node
I20251029 16:34:08.717646 8300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 16:34:08.719365 8300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:34:08.720582 8300 hybrid_clock.cc:648] HybridClock initialized: now 1761755648720544 us; error 43 us; skew 500 ppm
I20251029 16:34:08.720990 8300 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:34:08.723444 8300 webserver.cc:492] Webserver started at http://127.6.180.126:40759/ using document root <none> and password file <none>
I20251029 16:34:08.724413 8300 fs_manager.cc:362] Metadata directory not provided
I20251029 16:34:08.724576 8300 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:34:08.730724 8300 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20251029 16:34:08.734113 8320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:34:08.735771 8300 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251029 16:34:08.736016 8300 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/wal
uuid: "0442da13a653435dba87fc90d8e30d1f"
format_stamp: "Formatted at 2025-10-29 16:33:56 on dist-test-slave-l1bh"
I20251029 16:34:08.736785 8300 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:34:08.756587 8300 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 16:34:08.757531 8300 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:34:08.757822 8300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 16:34:08.779255 8300 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.126:36819
I20251029 16:34:08.779281 8371 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.126:36819 every 8 connection(s)
I20251029 16:34:08.781579 8300 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-0/data/info.pb
I20251029 16:34:08.785880 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 8300
I20251029 16:34:08.793299 8372 sys_catalog.cc:263] Verifying existing consensus state
I20251029 16:34:08.800595 8372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: Bootstrap starting.
I20251029 16:34:08.813640 8372 log.cc:826] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: Log is configured to *not* fsync() on all Append() calls
I20251029 16:34:08.843766 8372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=25 ignored=0} mutations{seen=20 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251029 16:34:08.844422 8372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: Bootstrap complete.
I20251029 16:34:08.850239 8372 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:08.851017 8372 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0442da13a653435dba87fc90d8e30d1f, State: Initialized, Role: FOLLOWER
I20251029 16:34:08.851713 8372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 3.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:08.852689 8377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:08.852975 8377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:08.853828 8372 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: configured and running, proceeding with master startup.
I20251029 16:34:08.868412 8392 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:08.868646 8392 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: loading cluster ID for follower catalog manager: success
I20251029 16:34:08.874266 8392 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: acquiring CA information for follower catalog manager: success
I20251029 16:34:08.877395 8392 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 16:34:08.879417 8300 master_runner.cc:186] Error getting master registration for 127.6.180.125:44441: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111),
I20251029 16:34:08.884891 8300 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:08.928651 7818 heartbeater.cc:344] Connected to a master server at 127.6.180.126:36819
I20251029 16:34:08.932927 8337 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b94ca437df08467bae40554a10450269" instance_seqno: 1761755639575696) as {username='slave'} at 127.6.180.67:54251; Asking this server to re-register.
I20251029 16:34:08.934252 7818 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:08.936724 8337 ts_manager.cc:194] Registered new tserver with Master: b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:09.115357 7525 heartbeater.cc:344] Connected to a master server at 127.6.180.126:36819
I20251029 16:34:09.117794 8337 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" instance_seqno: 1761755638480543) as {username='slave'} at 127.6.180.65:37895; Asking this server to re-register.
I20251029 16:34:09.118588 7525 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:09.120431 8337 ts_manager.cc:194] Registered new tserver with Master: 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551)
I20251029 16:34:09.188593 7676 heartbeater.cc:344] Connected to a master server at 127.6.180.126:36819
I20251029 16:34:09.191507 8337 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" instance_seqno: 1761755639071323) as {username='slave'} at 127.6.180.66:43267; Asking this server to re-register.
I20251029 16:34:09.192445 7676 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:09.194375 8337 ts_manager.cc:194] Registered new tserver with Master: 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:09.232470 8377 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:09.232800 8377 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:09.235131 8377 leader_election.cc:290] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:09.236035 8324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
W20251029 16:34:09.239109 8324 leader_election.cc:336] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:09.243348 8200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0442da13a653435dba87fc90d8e30d1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "29f9bab9976745478595681935778cac" is_pre_election: true
I20251029 16:34:09.243677 8200 raft_consensus.cc:2368] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0442da13a653435dba87fc90d8e30d1f for earlier term 4. Current term is 5.
I20251029 16:34:09.244521 8322 leader_election.cc:400] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 4 pre-election: Vote denied by peer 29f9bab9976745478595681935778cac (127.6.180.124:39271) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0442da13a653435dba87fc90d8e30d1f for earlier term 4. Current term is 5.
I20251029 16:34:09.244664 8322 leader_election.cc:403] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20251029 16:34:09.245002 8377 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 3 FOLLOWER]: Advancing to term 5
I20251029 16:34:09.249503 8377 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 29f9bab9976745478595681935778cac (127.6.180.124:39271) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0442da13a653435dba87fc90d8e30d1f for earlier term 4. Current term is 5.
I20251029 16:34:09.868691 8404 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:09.868901 8404 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:09.869697 8404 leader_election.cc:290] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
I20251029 16:34:09.870325 8200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0442da13a653435dba87fc90d8e30d1f" candidate_term: 6 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "29f9bab9976745478595681935778cac" is_pre_election: true
I20251029 16:34:09.870643 8200 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0442da13a653435dba87fc90d8e30d1f for term 6 because replica has last-logged OpId of term: 5 index: 42, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 34.
W20251029 16:34:09.872174 8324 leader_election.cc:336] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:09.872300 8324 leader_election.cc:304] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f; no voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:09.872665 8404 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251029 16:34:10.260569 8156 master_runner.cc:186] Error getting master registration for 127.6.180.125:44441: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111),
I20251029 16:34:10.260785 8156 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:10.489873 8413 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:10.490093 8413 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:10.490820 8413 leader_election.cc:290] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
I20251029 16:34:10.491583 8200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0442da13a653435dba87fc90d8e30d1f" candidate_term: 6 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "29f9bab9976745478595681935778cac" is_pre_election: true
I20251029 16:34:10.491899 8200 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0442da13a653435dba87fc90d8e30d1f for term 6 because replica has last-logged OpId of term: 5 index: 42, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 34.
W20251029 16:34:10.493103 8324 leader_election.cc:336] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:10.493211 8324 leader_election.cc:304] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f; no voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:10.493487 8413 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251029 16:34:10.800393 8416 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3155690d4bb54c97b5a1e7881608ae08)
I20251029 16:34:10.800594 8416 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:10.801335 8416 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
I20251029 16:34:10.802103 8347 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "0442da13a653435dba87fc90d8e30d1f" is_pre_election: true
I20251029 16:34:10.802601 8347 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f9bab9976745478595681935778cac in term 5.
I20251029 16:34:10.803331 8175 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac; no voters:
I20251029 16:34:10.803658 8416 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251029 16:34:10.803792 8416 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3155690d4bb54c97b5a1e7881608ae08)
I20251029 16:34:10.803901 8416 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 5 FOLLOWER]: Advancing to term 6
W20251029 16:34:10.804380 8177 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:10.806641 8416 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:10.807404 8416 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 election: Requested vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
I20251029 16:34:10.807847 8347 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "0442da13a653435dba87fc90d8e30d1f"
I20251029 16:34:10.808095 8347 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 5 FOLLOWER]: Advancing to term 6
W20251029 16:34:10.809101 8177 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:10.810487 8347 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f9bab9976745478595681935778cac in term 6.
I20251029 16:34:10.810945 8175 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac; no voters: 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:10.811316 8416 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 6 FOLLOWER]: Leader election won for term 6
I20251029 16:34:10.811758 8416 raft_consensus.cc:697] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 6 LEADER]: Becoming Leader. State: Replica: 29f9bab9976745478595681935778cac, State: Running, Role: LEADER
I20251029 16:34:10.812252 8416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 5.42, Last appended by leader: 42, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:10.814275 8419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f9bab9976745478595681935778cac. Latest consensus state: current_term: 6 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:10.814647 8419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: LEADER
I20251029 16:34:11.017338 6865 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskoTseiL/build/asan/bin/kudu pid 8156
I20251029 16:34:11.028537 8220 generic_service.cc:196] Checking for leaks (request via RPC)
I20251029 16:34:11.138772 6865 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoTseiL/build/asan/bin/kudu with pid 8156
W20251029 16:34:11.159554 7411 connection.cc:537] client connection to 127.6.180.124:39271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251029 16:34:11.834932 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.124:39271
--webserver_interface=127.6.180.124
--webserver_port=46685
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251029 16:34:12.211342 8432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:34:12.211692 8432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:34:12.211772 8432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:34:12.221014 8432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 16:34:12.221123 8432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:34:12.221232 8432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 16:34:12.221287 8432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 16:34:12.232014 8432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.124:39271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.180.124
--webserver_port=46685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:34:12.233750 8432 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:34:12.235754 8432 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 16:34:12.245054 8437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:12.245002 8438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:12.245117 8440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:34:12.245986 8432 server_base.cc:1047] running on GCE node
I20251029 16:34:12.246837 8432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 16:34:12.248785 8432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:34:12.250008 8432 hybrid_clock.cc:648] HybridClock initialized: now 1761755652249955 us; error 53 us; skew 500 ppm
I20251029 16:34:12.250411 8432 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:34:12.253000 8432 webserver.cc:492] Webserver started at http://127.6.180.124:46685/ using document root <none> and password file <none>
I20251029 16:34:12.253904 8432 fs_manager.cc:362] Metadata directory not provided
I20251029 16:34:12.254016 8432 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:34:12.260406 8432 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20251029 16:34:12.263646 8446 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:34:12.265398 8432 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251029 16:34:12.265599 8432 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
uuid: "29f9bab9976745478595681935778cac"
format_stamp: "Formatted at 2025-10-29 16:33:57 on dist-test-slave-l1bh"
I20251029 16:34:12.266535 8432 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:34:12.281576 8432 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 16:34:12.282483 8432 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:34:12.282765 8432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 16:34:12.303141 8432 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.124:39271
I20251029 16:34:12.303166 8497 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.124:39271 every 8 connection(s)
I20251029 16:34:12.305088 8432 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
I20251029 16:34:12.307314 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 8432
I20251029 16:34:12.314225 8498 sys_catalog.cc:263] Verifying existing consensus state
I20251029 16:34:12.321934 8498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Bootstrap starting.
I20251029 16:34:12.335633 8498 log.cc:826] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Log is configured to *not* fsync() on all Append() calls
I20251029 16:34:12.369843 8498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Bootstrap replayed 1/1 log segments. Stats: ops{read=43 overwritten=0 applied=41 ignored=0} inserts{seen=30 ignored=0} mutations{seen=25 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251029 16:34:12.370507 8498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Bootstrap complete.
I20251029 16:34:12.376736 8498 raft_consensus.cc:359] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 6 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:12.379673 8498 raft_consensus.cc:740] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f9bab9976745478595681935778cac, State: Initialized, Role: FOLLOWER
I20251029 16:34:12.380483 8498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 6.43, Last appended by leader: 43, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:12.381299 8504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:12.381561 8504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:12.382498 8498 sys_catalog.cc:565] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: configured and running, proceeding with master startup.
I20251029 16:34:12.408044 8518 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:12.408196 8518 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: loading cluster ID for follower catalog manager: success
I20251029 16:34:12.412894 8518 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: acquiring CA information for follower catalog manager: success
I20251029 16:34:12.416440 8518 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 16:34:12.419195 8432 master_runner.cc:186] Error getting master registration for 127.6.180.125:44441: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111),
I20251029 16:34:12.423071 8432 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:12.463570 7817 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:34:12.466982 8462 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b94ca437df08467bae40554a10450269" instance_seqno: 1761755639575696) as {username='slave'} at 127.6.180.67:33379; Asking this server to re-register.
I20251029 16:34:12.467984 7817 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:12.470178 8462 ts_manager.cc:194] Registered new tserver with Master: b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:12.474651 8523 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:12.474906 8523 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:12.475632 8523 leader_election.cc:290] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:12.478616 8324 leader_election.cc:336] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:12.484189 8473 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0442da13a653435dba87fc90d8e30d1f" candidate_term: 7 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "29f9bab9976745478595681935778cac" is_pre_election: true
I20251029 16:34:12.485005 8473 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0442da13a653435dba87fc90d8e30d1f for term 7 because replica has last-logged OpId of term: 6 index: 43, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 34.
I20251029 16:34:12.485915 8322 leader_election.cc:304] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f; no voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:12.486327 8523 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251029 16:34:12.671883 7524 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:34:12.673404 8462 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" instance_seqno: 1761755638480543) as {username='slave'} at 127.6.180.65:47565; Asking this server to re-register.
I20251029 16:34:12.673993 7524 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:12.675518 8462 ts_manager.cc:194] Registered new tserver with Master: 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551)
I20251029 16:34:12.726969 7675 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:34:12.729025 8462 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" instance_seqno: 1761755639071323) as {username='slave'} at 127.6.180.66:60143; Asking this server to re-register.
I20251029 16:34:12.729825 7675 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:12.732115 8462 ts_manager.cc:194] Registered new tserver with Master: 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:12.909418 8527 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:12.909730 8527 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:12.911592 8527 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
I20251029 16:34:12.912268 8347 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 7 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "0442da13a653435dba87fc90d8e30d1f" is_pre_election: true
W20251029 16:34:12.912336 8450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:12.912576 8347 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f9bab9976745478595681935778cac in term 6.
I20251029 16:34:12.913192 8448 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac; no voters:
I20251029 16:34:12.913569 8527 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251029 16:34:12.913751 8527 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 16:34:12.913861 8527 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 6 FOLLOWER]: Advancing to term 7
W20251029 16:34:12.914004 8450 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:12.917645 8527 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:12.918282 8527 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 7 election: Requested vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
I20251029 16:34:12.918862 8347 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 7 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "0442da13a653435dba87fc90d8e30d1f"
I20251029 16:34:12.919103 8347 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 6 FOLLOWER]: Advancing to term 7
W20251029 16:34:12.919723 8450 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:12.921357 8347 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f9bab9976745478595681935778cac in term 7.
I20251029 16:34:12.921793 8448 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac; no voters: 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:12.922107 8527 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 7 FOLLOWER]: Leader election won for term 7
I20251029 16:34:12.922468 8527 raft_consensus.cc:697] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 7 LEADER]: Becoming Leader. State: Replica: 29f9bab9976745478595681935778cac, State: Running, Role: LEADER
I20251029 16:34:12.922961 8527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 6.43, Last appended by leader: 43, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:12.925964 8529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f9bab9976745478595681935778cac. Latest consensus state: current_term: 7 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:12.926234 8529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: LEADER
I20251029 16:34:13.316913 8347 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Refusing update from remote peer 29f9bab9976745478595681935778cac: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 7 index: 44. (index mismatch)
I20251029 16:34:13.317884 8529 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 34, Time since last communication: 0.000s
I20251029 16:34:13.330089 8543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:13.330556 8543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: This master's current role is: FOLLOWER
W20251029 16:34:13.330664 8450 consensus_peers.cc:597] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac -> Peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Couldn't send request to peer 3155690d4bb54c97b5a1e7881608ae08. Status: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251029 16:34:13.330557 8534 mvcc.cc:204] Tried to move back new op lower bound from 7216151124009340928 to 7216151108819398656. Current Snapshot: MvccSnapshot[applied={T|T < 7216151124009340928 or (T in {7216151124009340928})}]
I20251029 16:34:13.330960 8533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f9bab9976745478595681935778cac. Latest consensus state: current_term: 7 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:13.331058 8537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:13.331314 8533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:13.331377 8537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:13.333351 8529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:13.333631 8529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: LEADER
I20251029 16:34:13.333825 8529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:13.334005 8529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: LEADER
I20251029 16:34:13.334412 8533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:13.334627 8533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:13.334446 8537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:13.334797 8537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:13.340521 8531 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251029 16:34:13.343391 8531 catalog_manager.cc:679] Loaded metadata for table default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe]
I20251029 16:34:13.343832 8531 catalog_manager.cc:679] Loaded metadata for table default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab]
I20251029 16:34:13.344195 8531 catalog_manager.cc:679] Loaded metadata for table default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c]
I20251029 16:34:13.344534 8531 catalog_manager.cc:679] Loaded metadata for table default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0]
I20251029 16:34:13.344965 8531 catalog_manager.cc:679] Loaded metadata for table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]
I20251029 16:34:13.345536 8531 catalog_manager.cc:679] Loaded metadata for table default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b]
I20251029 16:34:13.346109 8531 catalog_manager.cc:679] Loaded metadata for table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]
I20251029 16:34:13.346561 8531 catalog_manager.cc:679] Loaded metadata for table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]
I20251029 16:34:13.346895 8531 catalog_manager.cc:679] Loaded metadata for table default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6]
I20251029 16:34:13.348834 8531 tablet_loader.cc:96] loaded metadata for tablet 167c0aa33db046408bbc9bcd0844d152 (table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80])
I20251029 16:34:13.349380 8531 tablet_loader.cc:96] loaded metadata for tablet 28e1452396924512ac30d309085628ee (table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f])
I20251029 16:34:13.349695 8531 tablet_loader.cc:96] loaded metadata for tablet 41ad6aef032040f2933152a36de8d120 (table default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c])
I20251029 16:34:13.350131 8531 tablet_loader.cc:96] loaded metadata for tablet 4af807ac4f78451592a0a6788af12044 (table default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6])
I20251029 16:34:13.350596 8531 tablet_loader.cc:96] loaded metadata for tablet 8197c292e24348e1b633ba29e22a0bec (table default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0])
I20251029 16:34:13.351326 8531 tablet_loader.cc:96] loaded metadata for tablet a6d6b65d23ce4b84aab753d124d9e1cf (table default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab])
I20251029 16:34:13.351747 8531 tablet_loader.cc:96] loaded metadata for tablet acbbc2208b744b6c8ce5ea0fe699d29e (table default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe])
I20251029 16:34:13.352080 8531 tablet_loader.cc:96] loaded metadata for tablet af86f1498654491da54d0da20774edd5 (table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e])
I20251029 16:34:13.352694 8531 tablet_loader.cc:96] loaded metadata for tablet f5e3fd53975145e9acbc9dff7a2b0cb9 (table default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b])
I20251029 16:34:13.352988 8531 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251029 16:34:13.354182 8531 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:13.354272 8531 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251029 16:34:13.355902 8531 catalog_manager.cc:1514] Loading token signing keys...
I20251029 16:34:13.356974 8531 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Loaded TSK: 0
I20251029 16:34:13.357950 8531 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251029 16:34:13.358948 8531 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251029 16:34:13.359841 8531 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251029 16:34:13.385486 8463 master_service.cc:954] ReplaceTablet: received request to replace tablet 6a78066ce12c4092b69722de857c7a2a from {username='slave'} at 127.0.0.1:53544
I20251029 16:34:13.385617 8461 master_service.cc:954] ReplaceTablet: received request to replace tablet 1ddd091e76844d0cabcd2ea2ffe767cc from {username='slave'} at 127.0.0.1:53544
I20251029 16:34:13.386251 8460 master_service.cc:954] ReplaceTablet: received request to replace tablet 4233e0df604c4ff8a7e60528788d7d07 from {username='slave'} at 127.0.0.1:53544
I20251029 16:34:13.386659 8457 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53544:
name: "default.Table_70f60abb54374bb5a1a1475de5feec56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:13.387073 8454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53544:
table { table_name: "default.Table_e6381c7d53654326b15714d30d7eba93" } modify_external_catalogs: true
I20251029 16:34:13.387413 8454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53544:
table { table_name: "default.Table_e6381c7d53654326b15714d30d7eba93" } modify_external_catalogs: true
I20251029 16:34:13.387733 8462 master_service.cc:954] ReplaceTablet: received request to replace tablet 82b72b5cd6a1496eb09ac78395a285a1 from {username='slave'} at 127.0.0.1:53544
I20251029 16:34:13.387784 8459 master_service.cc:954] ReplaceTablet: received request to replace tablet 41ad6aef032040f2933152a36de8d120 from {username='slave'} at 127.0.0.1:53544
I20251029 16:34:13.388458 8463 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53544:
table { table_name: "default.Table_789f901f60d34caf9ddd2eb0e44c9dd2" } modify_external_catalogs: true
I20251029 16:34:13.388494 8460 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53544:
table { table_name: "default.Table_0982a1f0341d4f3f9e7f27ed71278b4d" } modify_external_catalogs: true
I20251029 16:34:13.388590 8455 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53544:
table { table_name: "default.Table_207ddcecc1c54b149e04a4d6f8190acb" } modify_external_catalogs: true
I20251029 16:34:13.388748 8463 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53544:
table { table_name: "default.Table_789f901f60d34caf9ddd2eb0e44c9dd2" } modify_external_catalogs: true
I20251029 16:34:13.388806 8455 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53544:
table { table_name: "default.Table_207ddcecc1c54b149e04a4d6f8190acb" } modify_external_catalogs: true
I20251029 16:34:13.389008 8460 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53544:
table { table_name: "default.Table_0982a1f0341d4f3f9e7f27ed71278b4d" } modify_external_catalogs: true
I20251029 16:34:13.389300 8462 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53544:
name: "default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:13.389930 8456 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53544:
name: "default.Table_700613fee8534a63b231420a5e166fb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:13.390064 8458 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53544:
name: "default.Table_ca14635728c14a7fbd4bce9c90ec208d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:13.391034 8461 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53544:
name: "default.Table_4bedb57357b24f78a75d1f362473ceff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251029 16:34:13.396754 8517 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1761755639, eventType=CREATE_TABLE, dbName=default, tableName=table_b513b87e8a1943eea3e24140ede4921d, message={"server":"","servicePrincipal":"","db":"default","table":"table_b513b87e8a1943eea3e24140ede4921d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b513b87e8a1943eea3e24140ede4921d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761755639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/warehouse/table_b513b87e8a1943eea3e24140ede4921d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b513b87e8a1943eea3e24140ede4921d\",\"kudu.table_id\":\"1398ac3c2f22427b96af2e9ff15e62d4\",\"transient_lastDdlTime\":\"1761755639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"415c0df9d5524c9a82ceee0f02442bac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761755640,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761755640, eventType=CREATE_TABLE, dbName=default, tableName=table_4d4ea90e32c7496a91ee99d2ca5a1f01, message={"server":"","servicePrincipal":"","db":"default","table":"table_4d4ea90e32c7496a91ee99d2ca5a1f01","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_4d4ea90e32c7496a91ee99d2ca5a1f01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761755640},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/warehouse/table_4d4ea90e32c7496a91ee99d2ca5a1f01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_4d4ea90e32c7496a91ee99d2ca5a1f01\",\"kudu.table_id\":\"d31229edb9834ddab88752bc8d367e80\",\"transient_lastDdlTime\":\"1761755640\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"415c0df9d5524c9a82ceee0f02442bac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761755640,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251029 16:34:13.397153 8459 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Sending DeleteTablet for 3 replicas of tablet 41ad6aef032040f2933152a36de8d120
I20251029 16:34:13.400633 8459 catalog_manager.cc:6719] ReplaceTablet: tablet 41ad6aef032040f2933152a36de8d120 deleted and replaced by tablet 17d7be76469746ea80fd24a579212d0a
I20251029 16:34:13.401333 8459 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53544:
table { table_name: "default.Table_4d4ea90e32c7496a91ee99d2ca5a1f01" } new_table_name: "default.Table_2ce7bbbb663d4ced9e2a662c79543a9e" modify_external_catalogs: true
@ 0x7f3dfcf81dcd google::LogMessage::Fail() at ??:0
@ 0x7f3dfcf85b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f3dfcf817cc google::LogMessage::Flush() at ??:0
@ 0x7f3dfcf81a11 google::LogMessage::~LogMessage() at ??:0
I20251029 16:34:13.412626 7457 tablet_service.cc:1552] Processing DeleteTablet for tablet 41ad6aef032040f2933152a36de8d120 with delete_type TABLET_DATA_DELETED (replaced by tablet 17d7be76469746ea80fd24a579212d0a) from {username='slave'} at 127.0.0.1:46880
@ 0x7f3e13943b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251029 16:34:13.420648 7609 tablet_service.cc:1552] Processing DeleteTablet for tablet 41ad6aef032040f2933152a36de8d120 with delete_type TABLET_DATA_DELETED (replaced by tablet 17d7be76469746ea80fd24a579212d0a) from {username='slave'} at 127.0.0.1:47614
I20251029 16:34:13.422921 7458 tablet_service.cc:1505] Processing CreateTablet for tablet 17d7be76469746ea80fd24a579212d0a (DEFAULT_TABLE table=default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:13.423502 7458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d7be76469746ea80fd24a579212d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:13.424669 8557 tablet_replica.cc:333] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46: stopping tablet replica
I20251029 16:34:13.424844 8557 raft_consensus.cc:2243] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:13.425140 8557 raft_consensus.cc:2272] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:13.425871 8557 ts_tablet_manager.cc:1916] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f3e13942411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251029 16:34:13.430181 7751 tablet_service.cc:1552] Processing DeleteTablet for tablet 41ad6aef032040f2933152a36de8d120 with delete_type TABLET_DATA_DELETED (replaced by tablet 17d7be76469746ea80fd24a579212d0a) from {username='slave'} at 127.0.0.1:38984
I20251029 16:34:13.430745 7609 tablet_service.cc:1505] Processing CreateTablet for tablet 17d7be76469746ea80fd24a579212d0a (DEFAULT_TABLE table=default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:13.431200 7609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d7be76469746ea80fd24a579212d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:13.432895 7750 tablet_service.cc:1505] Processing CreateTablet for tablet 17d7be76469746ea80fd24a579212d0a (DEFAULT_TABLE table=default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 16:34:13.433091 8557 ts_tablet_manager.cc:1929] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 16:34:13.433221 8557 log.cc:1199] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/wal/wals/41ad6aef032040f2933152a36de8d120
I20251029 16:34:13.433491 7750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d7be76469746ea80fd24a579212d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 16:34:13.433718 8557 ts_tablet_manager.cc:1950] T 41ad6aef032040f2933152a36de8d120 P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting consensus metadata
I20251029 16:34:13.435132 8447 catalog_manager.cc:4985] TS 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): tablet 41ad6aef032040f2933152a36de8d120 (table default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c]) successfully deleted
@ 0x7f3e1394a08c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251029 16:34:13.442679 8563 tablet_bootstrap.cc:492] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269: Bootstrap starting.
I20251029 16:34:13.445372 8563 tablet_bootstrap.cc:654] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269: Neither blocks nor log segments found. Creating new log.
@ 0x7f3e13949f09 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
I20251029 16:34:13.453750 8561 tablet_bootstrap.cc:492] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46: Bootstrap starting.
@ 0x7f3dffcf5ee9 kudu::Thread::SuperviseThread() at ??:0
I20251029 16:34:13.458402 8565 tablet_bootstrap.cc:492] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c: Bootstrap starting.
I20251029 16:34:13.460089 8561 tablet_bootstrap.cc:654] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46: Neither blocks nor log segments found. Creating new log.
I20251029 16:34:13.461181 8565 tablet_bootstrap.cc:654] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c: Neither blocks nor log segments found. Creating new log.
@ 0x7f3e033006db start_thread at ??:0
@ 0x7f3dfbe3271f clone at ??:0
I20251029 16:34:13.471321 8561 tablet_bootstrap.cc:492] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46: No bootstrap required, opened a new log
I20251029 16:34:13.471539 8561 ts_tablet_manager.cc:1403] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.005s
I20251029 16:34:13.472618 8561 raft_consensus.cc:359] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:13.472882 8561 raft_consensus.cc:385] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:13.472992 8561 raft_consensus.cc:740] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec98b5d57094792a1aff2a1df4e7a46, State: Initialized, Role: FOLLOWER
I20251029 16:34:13.473716 8559 tablet_replica.cc:333] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269: stopping tablet replica
I20251029 16:34:13.473979 8559 raft_consensus.cc:2243] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:13.474277 8559 raft_consensus.cc:2272] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:13.474472 8561 consensus_queue.cc:260] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:13.474841 8532 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:13.475023 8532 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:13.475340 8559 ts_tablet_manager.cc:1916] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:13.475843 8532 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551), b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:13.479274 7477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d7be76469746ea80fd24a579212d0a" candidate_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7cbc2353a042bba0a6103565a2828c" is_pre_election: true
I20251029 16:34:13.479440 7770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d7be76469746ea80fd24a579212d0a" candidate_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94ca437df08467bae40554a10450269" is_pre_election: true
W20251029 16:34:13.480228 7564 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551): Illegal state: must be running to vote when last-logged opid is not known
W20251029 16:34:13.480710 7562 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Illegal state: must be running to vote when last-logged opid is not known
I20251029 16:34:13.480844 7562 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ec98b5d57094792a1aff2a1df4e7a46; no voters: 4a7cbc2353a042bba0a6103565a2828c, b94ca437df08467bae40554a10450269
I20251029 16:34:13.481415 8532 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 16:34:13.481649 8555 tablet_replica.cc:333] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c: stopping tablet replica
I20251029 16:34:13.481987 8555 raft_consensus.cc:2243] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [term 1 LEADER]: Raft consensus shutting down.
I20251029 16:34:13.482797 8555 raft_consensus.cc:2272] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:13.483739 8555 ts_tablet_manager.cc:1916] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:13.500626 8561 ts_tablet_manager.cc:1434] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46: Time spent starting tablet: real 0.029s user 0.014s sys 0.014s
I20251029 16:34:13.504714 8563 tablet_bootstrap.cc:492] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269: No bootstrap required, opened a new log
I20251029 16:34:13.504897 8563 ts_tablet_manager.cc:1403] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269: Time spent bootstrapping tablet: real 0.062s user 0.013s sys 0.022s
I20251029 16:34:13.505795 8563 raft_consensus.cc:359] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:13.506011 8563 raft_consensus.cc:385] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:13.506106 8563 raft_consensus.cc:740] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ca437df08467bae40554a10450269, State: Initialized, Role: FOLLOWER
I20251029 16:34:13.506386 8563 consensus_queue.cc:260] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:13.508538 8565 tablet_bootstrap.cc:492] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c: No bootstrap required, opened a new log
I20251029 16:34:13.508754 8565 ts_tablet_manager.cc:1403] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c: Time spent bootstrapping tablet: real 0.051s user 0.016s sys 0.020s
I20251029 16:34:13.509830 8565 raft_consensus.cc:359] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:13.510062 8565 raft_consensus.cc:385] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 16:34:13.510162 8565 raft_consensus.cc:740] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7cbc2353a042bba0a6103565a2828c, State: Initialized, Role: FOLLOWER
I20251029 16:34:13.510634 8563 ts_tablet_manager.cc:1434] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20251029 16:34:13.510525 8565 consensus_queue.cc:260] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:13.511133 8565 ts_tablet_manager.cc:1434] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 16:34:13.519302 8559 ts_tablet_manager.cc:1929] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 16:34:13.520677 8559 log.cc:1199] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/wal/wals/41ad6aef032040f2933152a36de8d120
I20251029 16:34:13.521140 8559 ts_tablet_manager.cc:1950] T 41ad6aef032040f2933152a36de8d120 P b94ca437df08467bae40554a10450269: Deleting consensus metadata
I20251029 16:34:13.523569 8555 ts_tablet_manager.cc:1929] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 16:34:13.523670 8555 log.cc:1199] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-0/wal/wals/41ad6aef032040f2933152a36de8d120
I20251029 16:34:13.524052 8555 ts_tablet_manager.cc:1950] T 41ad6aef032040f2933152a36de8d120 P 4a7cbc2353a042bba0a6103565a2828c: Deleting consensus metadata
W20251029 16:34:13.683806 7413 connection.cc:537] server connection from 127.0.0.1:46880 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:13.683743 7704 connection.cc:537] server connection from 127.0.0.1:38984 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:13.683887 7705 connection.cc:537] client connection to 127.6.180.124:39271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:13.684406 7825 connection.cc:537] client connection to 127.6.180.124:39271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:13.684602 7563 connection.cc:537] client connection to 127.6.180.124:39271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:13.685087 7411 connection.cc:537] client connection to 127.6.180.124:39271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:13.685158 7825 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.6.180.124:39271)
W20251029 16:34:13.685792 7825 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.180.124:39271)
W20251029 16:34:13.686340 7825 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.6.180.124:39271)
W20251029 16:34:13.687466 7825 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.6.180.124:39271)
I20251029 16:34:13.808810 8295 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:13.809026 8295 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:13.809976 8295 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:13.810416 7477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d7be76469746ea80fd24a579212d0a" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7cbc2353a042bba0a6103565a2828c" is_pre_election: true
I20251029 16:34:13.810534 7628 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d7be76469746ea80fd24a579212d0a" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" is_pre_election: true
I20251029 16:34:13.810685 7477 raft_consensus.cc:2468] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b94ca437df08467bae40554a10450269 in term 0.
I20251029 16:34:13.810758 7628 raft_consensus.cc:2468] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b94ca437df08467bae40554a10450269 in term 0.
I20251029 16:34:13.811165 7706 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c, b94ca437df08467bae40554a10450269; no voters:
I20251029 16:34:13.811436 8295 raft_consensus.cc:2804] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 16:34:13.811533 8295 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 16:34:13.811619 8295 raft_consensus.cc:3060] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:13.813658 8295 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:13.814335 8295 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 election: Requested vote from peers 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:13.814935 7477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d7be76469746ea80fd24a579212d0a" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7cbc2353a042bba0a6103565a2828c"
I20251029 16:34:13.814954 7628 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d7be76469746ea80fd24a579212d0a" candidate_uuid: "b94ca437df08467bae40554a10450269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec98b5d57094792a1aff2a1df4e7a46"
I20251029 16:34:13.815161 7477 raft_consensus.cc:3060] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:13.815181 7628 raft_consensus.cc:3060] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 0 FOLLOWER]: Advancing to term 1
I20251029 16:34:13.817593 7477 raft_consensus.cc:2468] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94ca437df08467bae40554a10450269 in term 1.
I20251029 16:34:13.817646 7628 raft_consensus.cc:2468] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94ca437df08467bae40554a10450269 in term 1.
I20251029 16:34:13.818051 7706 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c, b94ca437df08467bae40554a10450269; no voters:
I20251029 16:34:13.818358 8295 raft_consensus.cc:2804] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 16:34:13.818493 8295 raft_consensus.cc:697] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [term 1 LEADER]: Becoming Leader. State: Replica: b94ca437df08467bae40554a10450269, State: Running, Role: LEADER
I20251029 16:34:13.819039 8295 consensus_queue.cc:237] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:13.887897 8300 master_runner.cc:186] Error getting master registration for 127.6.180.125:44441: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111),
I20251029 16:34:13.888095 8300 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:13.912746 8580 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 29f9bab9976745478595681935778cac)
I20251029 16:34:13.912938 8580 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:13.913610 8580 leader_election.cc:290] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:13.915278 8324 leader_election.cc:336] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
W20251029 16:34:13.915463 8322 leader_election.cc:336] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:13.915625 8322 leader_election.cc:304] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f; no voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:13.915884 8580 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251029 16:34:14.657171 8585 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 29f9bab9976745478595681935778cac)
I20251029 16:34:14.657359 8585 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:14.657996 8585 leader_election.cc:290] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:14.658926 8324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20251029 16:34:14.660426 8324 leader_election.cc:336] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
W20251029 16:34:14.660666 8322 leader_election.cc:336] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:14.660791 8322 leader_election.cc:304] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f; no voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:14.661013 8585 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251029 16:34:14.980131 8295 consensus_queue.cc:1048] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:14.988624 8295 consensus_queue.cc:1048] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 16:34:15.164474 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.125:44441
--webserver_interface=127.6.180.125
--webserver_port=41231
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251029 16:34:15.527369 8601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:34:15.527734 8601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:34:15.527878 8601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:34:15.536799 8601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 16:34:15.536918 8601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:34:15.536999 8601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 16:34:15.537078 8601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 16:34:15.547595 8601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.125:44441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.180.125
--webserver_port=41231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:34:15.549429 8601 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:34:15.551365 8601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 16:34:15.560254 8612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:15.560261 8610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:15.560300 8609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:34:15.561582 8601 server_base.cc:1047] running on GCE node
I20251029 16:34:15.562440 8601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 16:34:15.564141 8601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:34:15.565351 8601 hybrid_clock.cc:648] HybridClock initialized: now 1761755655565306 us; error 39 us; skew 500 ppm
I20251029 16:34:15.565815 8601 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:34:15.568327 8601 webserver.cc:492] Webserver started at http://127.6.180.125:41231/ using document root <none> and password file <none>
I20251029 16:34:15.569089 8601 fs_manager.cc:362] Metadata directory not provided
I20251029 16:34:15.569192 8601 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:34:15.575134 8601 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20251029 16:34:15.578701 8618 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:34:15.580348 8601 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251029 16:34:15.580538 8601 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
uuid: "3155690d4bb54c97b5a1e7881608ae08"
format_stamp: "Formatted at 2025-10-29 16:33:56 on dist-test-slave-l1bh"
I20251029 16:34:15.581336 8601 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:34:15.593912 8623 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 29f9bab9976745478595681935778cac)
I20251029 16:34:15.594122 8623 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:15.594821 8623 leader_election.cc:290] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:15.608788 8322 leader_election.cc:336] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
W20251029 16:34:15.609356 8324 leader_election.cc:336] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:15.609498 8324 leader_election.cc:304] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f; no voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:15.609822 8623 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251029 16:34:15.614023 8601 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 16:34:15.614974 8601 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:34:15.615274 8601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 16:34:15.636884 8601 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.125:44441
I20251029 16:34:15.636904 8674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.125:44441 every 8 connection(s)
I20251029 16:34:15.638677 8601 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
I20251029 16:34:15.646556 8675 sys_catalog.cc:263] Verifying existing consensus state
I20251029 16:34:15.647178 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 8601
I20251029 16:34:15.651244 8675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Bootstrap starting.
I20251029 16:34:15.667596 8675 log.cc:826] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Log is configured to *not* fsync() on all Append() calls
I20251029 16:34:15.698838 8675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=42 ignored=0} inserts{seen=30 ignored=0} mutations{seen=30 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251029 16:34:15.699472 8675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Bootstrap complete.
I20251029 16:34:15.704919 8675 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:15.705752 8675 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3155690d4bb54c97b5a1e7881608ae08, State: Initialized, Role: FOLLOWER
I20251029 16:34:15.706449 8675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:15.707298 8680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:15.707604 8680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:15.708616 8675 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: configured and running, proceeding with master startup.
I20251029 16:34:15.726722 8694 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:15.726831 8694 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: loading cluster ID for follower catalog manager: success
I20251029 16:34:15.734539 8694 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: acquiring CA information for follower catalog manager: success
I20251029 16:34:15.737881 8694 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 16:34:15.749605 8601 master_runner.cc:186] Error getting master registration for 127.6.180.124:39271: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111),
I20251029 16:34:15.755573 8601 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:15.817725 7819 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:34:15.821051 8638 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b94ca437df08467bae40554a10450269" instance_seqno: 1761755639575696) as {username='slave'} at 127.6.180.67:32843; Asking this server to re-register.
I20251029 16:34:15.822072 7819 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:15.824263 8638 ts_manager.cc:194] Registered new tserver with Master: b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:15.996767 7526 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:34:15.998795 8638 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" instance_seqno: 1761755638480543) as {username='slave'} at 127.6.180.65:36183; Asking this server to re-register.
I20251029 16:34:15.999764 7526 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:16.001647 8638 ts_manager.cc:194] Registered new tserver with Master: 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551)
I20251029 16:34:16.004105 7677 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:34:16.005836 8638 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" instance_seqno: 1761755639071323) as {username='slave'} at 127.6.180.66:43269; Asking this server to re-register.
I20251029 16:34:16.006505 7677 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:16.008035 8638 ts_manager.cc:194] Registered new tserver with Master: 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:16.117816 8680 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:16.118132 8680 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:16.120548 8680 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
I20251029 16:34:16.121150 8346 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3155690d4bb54c97b5a1e7881608ae08" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "0442da13a653435dba87fc90d8e30d1f" is_pre_election: true
I20251029 16:34:16.121450 8346 raft_consensus.cc:2368] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3155690d4bb54c97b5a1e7881608ae08 for earlier term 6. Current term is 7.
W20251029 16:34:16.121351 8620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:16.127202 8620 leader_election.cc:400] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3155690d4bb54c97b5a1e7881608ae08 for earlier term 6. Current term is 7.
I20251029 16:34:16.127341 8620 leader_election.cc:403] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20251029 16:34:16.127705 8680 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 5 FOLLOWER]: Advancing to term 7
W20251029 16:34:16.128289 8620 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:16.131771 8680 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3155690d4bb54c97b5a1e7881608ae08 for earlier term 6. Current term is 7.
I20251029 16:34:16.385957 8706 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 29f9bab9976745478595681935778cac)
I20251029 16:34:16.386175 8706 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:16.386976 8706 leader_election.cc:290] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:16.389308 8322 leader_election.cc:336] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:16.393363 8649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0442da13a653435dba87fc90d8e30d1f" candidate_term: 8 candidate_status { last_received { term: 7 index: 46 } } ignore_live_leader: false dest_uuid: "3155690d4bb54c97b5a1e7881608ae08" is_pre_election: true
I20251029 16:34:16.393857 8649 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0442da13a653435dba87fc90d8e30d1f in term 7.
I20251029 16:34:16.394454 8324 leader_election.cc:304] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08; no voters: 29f9bab9976745478595681935778cac
I20251029 16:34:16.394793 8706 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251029 16:34:16.394897 8706 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 29f9bab9976745478595681935778cac)
I20251029 16:34:16.395004 8706 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 7 FOLLOWER]: Advancing to term 8
I20251029 16:34:16.398170 8706 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:16.398855 8706 leader_election.cc:290] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 election: Requested vote from peers 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
I20251029 16:34:16.399508 8649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0442da13a653435dba87fc90d8e30d1f" candidate_term: 8 candidate_status { last_received { term: 7 index: 46 } } ignore_live_leader: false dest_uuid: "3155690d4bb54c97b5a1e7881608ae08"
I20251029 16:34:16.399755 8649 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 7 FOLLOWER]: Advancing to term 8
W20251029 16:34:16.400311 8322 leader_election.cc:336] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:16.402011 8649 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0442da13a653435dba87fc90d8e30d1f in term 8.
I20251029 16:34:16.402508 8324 leader_election.cc:304] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08; no voters: 29f9bab9976745478595681935778cac
I20251029 16:34:16.402837 8706 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 8 FOLLOWER]: Leader election won for term 8
I20251029 16:34:16.403254 8706 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [term 8 LEADER]: Becoming Leader. State: Replica: 0442da13a653435dba87fc90d8e30d1f, State: Running, Role: LEADER
I20251029 16:34:16.403716 8706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 7.46, Last appended by leader: 46, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:16.405448 8710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0442da13a653435dba87fc90d8e30d1f. Latest consensus state: current_term: 8 leader_uuid: "0442da13a653435dba87fc90d8e30d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:16.405670 8710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [sys.catalog]: This master's current role is: LEADER
I20251029 16:34:16.406294 8712 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251029 16:34:16.409051 8712 catalog_manager.cc:679] Loaded metadata for table default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe]
I20251029 16:34:16.409550 8712 catalog_manager.cc:679] Loaded metadata for table default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab]
I20251029 16:34:16.409953 8712 catalog_manager.cc:679] Loaded metadata for table default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c]
I20251029 16:34:16.410322 8712 catalog_manager.cc:679] Loaded metadata for table default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0]
I20251029 16:34:16.410656 8712 catalog_manager.cc:679] Loaded metadata for table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]
I20251029 16:34:16.410984 8712 catalog_manager.cc:679] Loaded metadata for table default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b]
I20251029 16:34:16.411305 8712 catalog_manager.cc:679] Loaded metadata for table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]
I20251029 16:34:16.411671 8712 catalog_manager.cc:679] Loaded metadata for table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]
I20251029 16:34:16.412111 8712 catalog_manager.cc:679] Loaded metadata for table default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6]
I20251029 16:34:16.414044 8712 tablet_loader.cc:96] loaded metadata for tablet 167c0aa33db046408bbc9bcd0844d152 (table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80])
I20251029 16:34:16.414470 8712 tablet_loader.cc:96] loaded metadata for tablet 17d7be76469746ea80fd24a579212d0a (table default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c])
I20251029 16:34:16.414920 8712 tablet_loader.cc:96] loaded metadata for tablet 28e1452396924512ac30d309085628ee (table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f])
I20251029 16:34:16.415771 8712 tablet_loader.cc:96] loaded metadata for tablet 4af807ac4f78451592a0a6788af12044 (table default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6])
I20251029 16:34:16.416837 8712 tablet_loader.cc:96] loaded metadata for tablet 8197c292e24348e1b633ba29e22a0bec (table default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0])
I20251029 16:34:16.417697 8712 tablet_loader.cc:96] loaded metadata for tablet a6d6b65d23ce4b84aab753d124d9e1cf (table default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab])
I20251029 16:34:16.418110 8712 tablet_loader.cc:96] loaded metadata for tablet acbbc2208b744b6c8ce5ea0fe699d29e (table default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe])
I20251029 16:34:16.418356 8712 tablet_loader.cc:96] loaded metadata for tablet af86f1498654491da54d0da20774edd5 (table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e])
I20251029 16:34:16.418920 8712 tablet_loader.cc:96] loaded metadata for tablet f5e3fd53975145e9acbc9dff7a2b0cb9 (table default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b])
I20251029 16:34:16.419171 8712 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251029 16:34:16.420414 8712 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:16.420496 8712 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251029 16:34:16.421970 8712 catalog_manager.cc:1514] Loading token signing keys...
I20251029 16:34:16.422945 8712 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f: Loaded TSK: 0
I20251029 16:34:16.423882 8712 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251029 16:34:16.424914 8712 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251029 16:34:16.425853 8712 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251029 16:34:16.500387 8335 master_service.cc:954] ReplaceTablet: received request to replace tablet af86f1498654491da54d0da20774edd5 from {username='slave'} at 127.0.0.1:57736
I20251029 16:34:16.500699 8336 master_service.cc:954] ReplaceTablet: received request to replace tablet a6d6b65d23ce4b84aab753d124d9e1cf from {username='slave'} at 127.0.0.1:57736
I20251029 16:34:16.501056 8334 master_service.cc:954] ReplaceTablet: received request to replace tablet e1ac79eb37644b3e8e02681941c20091 from {username='slave'} at 127.0.0.1:57736
I20251029 16:34:16.504184 8332 master_service.cc:954] ReplaceTablet: received request to replace tablet 4233e0df604c4ff8a7e60528788d7d07 from {username='slave'} at 127.0.0.1:57736
I20251029 16:34:16.504654 8333 master_service.cc:954] ReplaceTablet: received request to replace tablet a6d6b65d23ce4b84aab753d124d9e1cf from {username='slave'} at 127.0.0.1:57736
I20251029 16:34:16.505578 8337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57736:
name: "default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:16.505941 8334 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57736:
table { table_name: "default.Table_dfc244c581534b678f83267a7d10b58c" } new_table_name: "default.Table_35daf50c962249fd983f0e49e1f7960a" modify_external_catalogs: true
I20251029 16:34:16.503726 8331 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57736:
name: "default.Table_cf493c1cc4f44b069dba62ecf1ad3676"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:16.506389 8329 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57736:
name: "default.Table_75040a14be804518a951b603c808d402"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:16.508164 8330 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57736:
name: "default.Table_3a47f0a3057e46acaf74a5e33bffb30a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:16.505925 8328 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57736:
name: "default.Table_e4dfb9e8508e41669fcd08f8c79b24f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:16.508249 8332 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57736:
table { table_name: "default.Table_6129e3888a3346669ab7c8982593f438" } new_table_name: "default.Table_323e8fdbeff34a12add0ad9e4c97accd" modify_external_catalogs: true
W20251029 16:34:16.515995 8322 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f -> Peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Couldn't send request to peer 29f9bab9976745478595681935778cac. Status: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251029 16:34:16.516062 8649 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Refusing update from remote peer 0442da13a653435dba87fc90d8e30d1f: Log matching property violated. Preceding OpId in replica: term: 5 index: 42. Preceding OpId from leader: term: 8 index: 48. (index mismatch)
I20251029 16:34:16.520838 8710 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0442da13a653435dba87fc90d8e30d1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 42, Time since last communication: 0.000s
F20251029 16:34:16.523641 8391 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1761755639, eventType=CREATE_TABLE, dbName=default, tableName=table_b513b87e8a1943eea3e24140ede4921d, message={"server":"","servicePrincipal":"","db":"default","table":"table_b513b87e8a1943eea3e24140ede4921d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b513b87e8a1943eea3e24140ede4921d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761755639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/warehouse/table_b513b87e8a1943eea3e24140ede4921d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b513b87e8a1943eea3e24140ede4921d\",\"kudu.table_id\":\"1398ac3c2f22427b96af2e9ff15e62d4\",\"transient_lastDdlTime\":\"1761755639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"415c0df9d5524c9a82ceee0f02442bac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761755640,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761755640, eventType=CREATE_TABLE, dbName=default, tableName=table_4d4ea90e32c7496a91ee99d2ca5a1f01, message={"server":"","servicePrincipal":"","db":"default","table":"table_4d4ea90e32c7496a91ee99d2ca5a1f01","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_4d4ea90e32c7496a91ee99d2ca5a1f01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761755640},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/warehouse/table_4d4ea90e32c7496a91ee99d2ca5a1f01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_4d4ea90e32c7496a91ee99d2ca5a1f01\",\"kudu.table_id\":\"d31229edb9834ddab88752bc8d367e80\",\"transient_lastDdlTime\":\"1761755640\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"415c0df9d5524c9a82ceee0f02442bac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761755640,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f9a38a4ddcd google::LogMessage::Fail() at ??:0
@ 0x7f9a38a51b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f9a38a4d7cc google::LogMessage::Flush() at ??:0
@ 0x7f9a38a4da11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9a4f40fb64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f9a4f40e411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f9a4f41608c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f9a4f415f09 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
I20251029 16:34:16.565291 8719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "0442da13a653435dba87fc90d8e30d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:16.565663 8719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:16.566080 8680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0442da13a653435dba87fc90d8e30d1f. Latest consensus state: current_term: 8 leader_uuid: "0442da13a653435dba87fc90d8e30d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:16.566412 8680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:16.566289 8717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "0442da13a653435dba87fc90d8e30d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:16.566658 8717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:16.568300 8714 mvcc.cc:204] Tried to move back new op lower bound from 7216151156345868288 to 7216151145730805760. Current Snapshot: MvccSnapshot[applied={T|T < 7216151156345868288 or (T in {7216151156345868288})}]
@ 0x7f9a3b7c1ee9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9a3edcc6db start_thread at ??:0
@ 0x7f9a378fe71f clone at ??:0
I20251029 16:34:16.577891 8717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "0442da13a653435dba87fc90d8e30d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:16.578159 8717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
W20251029 16:34:16.778529 8622 connection.cc:537] server connection from 127.0.0.1:56946 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:16.779222 7825 connection.cc:537] client connection to 127.6.180.126:36819 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:16.780651 7825 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.6.180.126:36819)
W20251029 16:34:16.781086 7825 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.6.180.126:36819)
W20251029 16:34:16.781476 7825 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.180.126:36819)
W20251029 16:34:16.781795 7825 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.6.180.126:36819)
I20251029 16:34:17.001876 7525 heartbeater.cc:499] Master 127.6.180.126:36819 was elected leader, sending a full tablet report...
I20251029 16:34:17.003031 7676 heartbeater.cc:499] Master 127.6.180.126:36819 was elected leader, sending a full tablet report...
I20251029 16:34:17.073830 8717 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0442da13a653435dba87fc90d8e30d1f)
I20251029 16:34:17.074097 8717 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:17.074877 8717 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:17.077229 8620 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:17.077407 8620 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:17.077502 8620 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:17.077860 8717 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251029 16:34:17.762336 8739 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0442da13a653435dba87fc90d8e30d1f)
I20251029 16:34:17.762540 8739 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:17.763176 8739 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:17.765545 8620 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:17.765795 8620 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:17.765883 8620 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:17.766137 8739 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251029 16:34:18.325245 8744 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0442da13a653435dba87fc90d8e30d1f)
I20251029 16:34:18.325486 8744 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:18.326224 8744 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:18.328666 8620 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:18.328896 8620 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:18.328991 8620 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:18.329250 8744 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251029 16:34:18.664288 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.124:39271
--webserver_interface=127.6.180.124
--webserver_port=46685
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251029 16:34:19.019620 8748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:34:19.020047 8748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:34:19.020174 8748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:34:19.028985 8748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 16:34:19.029093 8748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:34:19.029163 8748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 16:34:19.029229 8748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 16:34:19.039462 8748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.124:39271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.180.124
--webserver_port=46685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:34:19.041289 8748 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:34:19.043211 8748 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 16:34:19.052058 8754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:19.052045 8753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:19.052430 8756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:34:19.053756 8748 server_base.cc:1047] running on GCE node
I20251029 16:34:19.054705 8748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 16:34:19.056392 8748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:34:19.057621 8748 hybrid_clock.cc:648] HybridClock initialized: now 1761755659057571 us; error 49 us; skew 500 ppm
I20251029 16:34:19.057991 8748 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:34:19.060766 8748 webserver.cc:492] Webserver started at http://127.6.180.124:46685/ using document root <none> and password file <none>
I20251029 16:34:19.061630 8748 fs_manager.cc:362] Metadata directory not provided
I20251029 16:34:19.061756 8748 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:34:19.067869 8748 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20251029 16:34:19.071246 8762 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:34:19.072917 8748 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251029 16:34:19.073107 8748 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
uuid: "29f9bab9976745478595681935778cac"
format_stamp: "Formatted at 2025-10-29 16:33:57 on dist-test-slave-l1bh"
I20251029 16:34:19.073859 8748 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:34:19.113632 8748 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 16:34:19.114565 8748 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:34:19.114815 8748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 16:34:19.137146 8748 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.124:39271
I20251029 16:34:19.137168 8815 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.124:39271 every 8 connection(s)
I20251029 16:34:19.138945 8748 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
I20251029 16:34:19.146488 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 8748
I20251029 16:34:19.146809 8816 sys_catalog.cc:263] Verifying existing consensus state
I20251029 16:34:19.151036 8816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Bootstrap starting.
I20251029 16:34:19.165665 8816 log.cc:826] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Log is configured to *not* fsync() on all Append() calls
I20251029 16:34:19.205322 8816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Bootstrap replayed 1/1 log segments. Stats: ops{read=46 overwritten=0 applied=46 ignored=0} inserts{seen=31 ignored=0} mutations{seen=32 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251029 16:34:19.205932 8816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Bootstrap complete.
I20251029 16:34:19.211496 8816 raft_consensus.cc:359] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:19.212378 8816 raft_consensus.cc:740] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f9bab9976745478595681935778cac, State: Initialized, Role: FOLLOWER
I20251029 16:34:19.213063 8816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 46, Last appended: 7.46, Last appended by leader: 46, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:19.213843 8820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:19.214107 8820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:19.215004 8816 sys_catalog.cc:565] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: configured and running, proceeding with master startup.
I20251029 16:34:19.221916 8834 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251029 16:34:19.228029 8748 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:34:19.232661 8748 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:19.324568 7817 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:34:19.327663 8779 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b94ca437df08467bae40554a10450269" instance_seqno: 1761755639575696) as {username='slave'} at 127.6.180.67:45773; Asking this server to re-register.
I20251029 16:34:19.328486 7817 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:19.330327 8779 ts_manager.cc:194] Registered new tserver with Master: b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:19.504747 7524 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:34:19.506415 8779 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" instance_seqno: 1761755638480543) as {username='slave'} at 127.6.180.65:60611; Asking this server to re-register.
I20251029 16:34:19.507112 7524 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:19.508648 8779 ts_manager.cc:194] Registered new tserver with Master: 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551)
I20251029 16:34:19.512040 7675 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:34:19.513485 8779 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" instance_seqno: 1761755639071323) as {username='slave'} at 127.6.180.66:33257; Asking this server to re-register.
I20251029 16:34:19.514058 7675 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:19.515344 8779 ts_manager.cc:194] Registered new tserver with Master: 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:19.709956 8820 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:19.710258 8820 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:19.712284 8820 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
W20251029 16:34:19.713196 8764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:19.714780 8764 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:34:19.719525 8648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 8 candidate_status { last_received { term: 7 index: 46 } } ignore_live_leader: false dest_uuid: "3155690d4bb54c97b5a1e7881608ae08" is_pre_election: true
I20251029 16:34:19.719861 8648 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 29f9bab9976745478595681935778cac in current term 8: Already voted for candidate 0442da13a653435dba87fc90d8e30d1f in this term.
I20251029 16:34:19.721351 8766 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f9bab9976745478595681935778cac; no voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:19.721767 8820 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 7 FOLLOWER]: Advancing to term 8
I20251029 16:34:19.726112 8820 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251029 16:34:19.798256 8842 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0442da13a653435dba87fc90d8e30d1f)
I20251029 16:34:19.798499 8842 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:19.799269 8842 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:19.802122 8620 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:34:19.805866 8790 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3155690d4bb54c97b5a1e7881608ae08" candidate_term: 9 candidate_status { last_received { term: 8 index: 49 } } ignore_live_leader: false dest_uuid: "29f9bab9976745478595681935778cac" is_pre_election: true
I20251029 16:34:19.806294 8790 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3155690d4bb54c97b5a1e7881608ae08 in term 8.
I20251029 16:34:19.806773 8620 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f
I20251029 16:34:19.807085 8842 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251029 16:34:19.807192 8842 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0442da13a653435dba87fc90d8e30d1f)
I20251029 16:34:19.807296 8842 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 8 FOLLOWER]: Advancing to term 9
I20251029 16:34:19.809584 8842 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:19.810228 8842 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 election: Requested vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
I20251029 16:34:19.811098 8790 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3155690d4bb54c97b5a1e7881608ae08" candidate_term: 9 candidate_status { last_received { term: 8 index: 49 } } ignore_live_leader: false dest_uuid: "29f9bab9976745478595681935778cac"
I20251029 16:34:19.811298 8790 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 8 FOLLOWER]: Advancing to term 9
W20251029 16:34:19.811542 8620 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:34:19.813498 8790 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3155690d4bb54c97b5a1e7881608ae08 in term 9.
I20251029 16:34:19.813926 8620 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f
I20251029 16:34:19.814260 8842 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 9 FOLLOWER]: Leader election won for term 9
I20251029 16:34:19.814672 8842 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 9 LEADER]: Becoming Leader. State: Replica: 3155690d4bb54c97b5a1e7881608ae08, State: Running, Role: LEADER
I20251029 16:34:19.815107 8842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 8.49, Last appended by leader: 49, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:19.816987 8846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3155690d4bb54c97b5a1e7881608ae08. Latest consensus state: current_term: 9 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:19.817324 8846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: LEADER
I20251029 16:34:19.817973 8848 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251029 16:34:19.821352 8848 catalog_manager.cc:679] Loaded metadata for table default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe]
I20251029 16:34:19.821997 8848 catalog_manager.cc:679] Loaded metadata for table default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab]
I20251029 16:34:19.822456 8848 catalog_manager.cc:679] Loaded metadata for table default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c]
I20251029 16:34:19.822819 8848 catalog_manager.cc:679] Loaded metadata for table default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0]
I20251029 16:34:19.823099 8848 catalog_manager.cc:679] Loaded metadata for table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]
I20251029 16:34:19.823410 8848 catalog_manager.cc:679] Loaded metadata for table default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b]
I20251029 16:34:19.823755 8848 catalog_manager.cc:679] Loaded metadata for table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]
I20251029 16:34:19.824111 8848 catalog_manager.cc:679] Loaded metadata for table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]
I20251029 16:34:19.824414 8848 catalog_manager.cc:679] Loaded metadata for table default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6]
I20251029 16:34:19.826292 8848 tablet_loader.cc:96] loaded metadata for tablet 167c0aa33db046408bbc9bcd0844d152 (table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80])
I20251029 16:34:19.826719 8848 tablet_loader.cc:96] loaded metadata for tablet 17d7be76469746ea80fd24a579212d0a (table default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c])
I20251029 16:34:19.827208 8848 tablet_loader.cc:96] loaded metadata for tablet 28e1452396924512ac30d309085628ee (table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f])
I20251029 16:34:19.827709 8848 tablet_loader.cc:96] loaded metadata for tablet 48947ffd82bf4768bed14e9bea56a60a (table default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab])
I20251029 16:34:19.827979 8848 tablet_loader.cc:96] loaded metadata for tablet 4af807ac4f78451592a0a6788af12044 (table default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6])
I20251029 16:34:19.828238 8848 tablet_loader.cc:96] loaded metadata for tablet 4f5e322159e6488d98400bfb9249522d (table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e])
I20251029 16:34:19.828648 8848 tablet_loader.cc:96] loaded metadata for tablet 8197c292e24348e1b633ba29e22a0bec (table default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0])
I20251029 16:34:19.829782 8848 tablet_loader.cc:96] loaded metadata for tablet acbbc2208b744b6c8ce5ea0fe699d29e (table default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe])
I20251029 16:34:19.830472 8848 tablet_loader.cc:96] loaded metadata for tablet f5e3fd53975145e9acbc9dff7a2b0cb9 (table default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b])
I20251029 16:34:19.830722 8848 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251029 16:34:19.832113 8848 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:19.832212 8848 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251029 16:34:19.833681 8848 catalog_manager.cc:1514] Loading token signing keys...
I20251029 16:34:19.834699 8848 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Loaded TSK: 0
I20251029 16:34:19.835563 8848 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251029 16:34:19.836601 8848 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251029 16:34:19.837576 8848 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251029 16:34:19.882789 8638 master_service.cc:954] ReplaceTablet: received request to replace tablet e1ac79eb37644b3e8e02681941c20091 from {username='slave'} at 127.0.0.1:56938
I20251029 16:34:19.882923 8639 master_service.cc:954] ReplaceTablet: received request to replace tablet 17d7be76469746ea80fd24a579212d0a from {username='slave'} at 127.0.0.1:56938
I20251029 16:34:19.883855 8635 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56938:
name: "default.Table_4bedb57357b24f78a75d1f362473ceff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:19.884630 8636 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56938:
table { table_name: "default.Table_4d4ea90e32c7496a91ee99d2ca5a1f01" } new_table_name: "default.Table_2ce7bbbb663d4ced9e2a662c79543a9e" modify_external_catalogs: true
I20251029 16:34:19.885183 8631 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56938:
table { table_name: "default.Table_e6381c7d53654326b15714d30d7eba93" } modify_external_catalogs: true
I20251029 16:34:19.885382 8637 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56938:
table { table_name: "default.Table_571d6fa0fe7e4782b8b0c692012fbd81" } new_table_name: "default.Table_ca38a68ba7574beab9c638e82db00946" modify_external_catalogs: true
I20251029 16:34:19.885476 8631 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56938:
table { table_name: "default.Table_e6381c7d53654326b15714d30d7eba93" } modify_external_catalogs: true
I20251029 16:34:19.883944 8634 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56938:
name: "default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:19.886008 8630 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56938:
table { table_name: "default.Table_207ddcecc1c54b149e04a4d6f8190acb" } modify_external_catalogs: true
I20251029 16:34:19.886296 8630 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56938:
table { table_name: "default.Table_207ddcecc1c54b149e04a4d6f8190acb" } modify_external_catalogs: true
I20251029 16:34:19.886673 8632 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56938:
table { table_name: "default.Table_789f901f60d34caf9ddd2eb0e44c9dd2" } modify_external_catalogs: true
I20251029 16:34:19.886873 8632 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56938:
table { table_name: "default.Table_789f901f60d34caf9ddd2eb0e44c9dd2" } modify_external_catalogs: true
I20251029 16:34:19.887095 8633 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56938:
table { table_name: "default.Table_0982a1f0341d4f3f9e7f27ed71278b4d" } modify_external_catalogs: true
I20251029 16:34:19.887363 8633 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56938:
table { table_name: "default.Table_0982a1f0341d4f3f9e7f27ed71278b4d" } modify_external_catalogs: true
I20251029 16:34:19.888063 8638 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56938:
name: "default.Table_cf493c1cc4f44b069dba62ecf1ad3676"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:19.891475 8790 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Refusing update from remote peer 3155690d4bb54c97b5a1e7881608ae08: Log matching property violated. Preceding OpId in replica: term: 7 index: 46. Preceding OpId from leader: term: 9 index: 51. (index mismatch)
I20251029 16:34:19.892572 8846 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 46, Time since last communication: 0.000s
W20251029 16:34:19.892884 8620 consensus_peers.cc:597] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 -> Peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Couldn't send request to peer 0442da13a653435dba87fc90d8e30d1f. Status: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20251029 16:34:19.893339 8693 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1761755639, eventType=CREATE_TABLE, dbName=default, tableName=table_b513b87e8a1943eea3e24140ede4921d, message={"server":"","servicePrincipal":"","db":"default","table":"table_b513b87e8a1943eea3e24140ede4921d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b513b87e8a1943eea3e24140ede4921d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761755639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/warehouse/table_b513b87e8a1943eea3e24140ede4921d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b513b87e8a1943eea3e24140ede4921d\",\"kudu.table_id\":\"1398ac3c2f22427b96af2e9ff15e62d4\",\"transient_lastDdlTime\":\"1761755639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"415c0df9d5524c9a82ceee0f02442bac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761755640,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761755640, eventType=CREATE_TABLE, dbName=default, tableName=table_4d4ea90e32c7496a91ee99d2ca5a1f01, message={"server":"","servicePrincipal":"","db":"default","table":"table_4d4ea90e32c7496a91ee99d2ca5a1f01","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_4d4ea90e32c7496a91ee99d2ca5a1f01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761755640},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/warehouse/table_4d4ea90e32c7496a91ee99d2ca5a1f01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_4d4ea90e32c7496a91ee99d2ca5a1f01\",\"kudu.table_id\":\"d31229edb9834ddab88752bc8d367e80\",\"transient_lastDdlTime\":\"1761755640\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"415c0df9d5524c9a82ceee0f02442bac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761755640,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f9fb47acdcd google::LogMessage::Fail() at ??:0
@ 0x7f9fb47b0b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f9fb47ac7cc google::LogMessage::Flush() at ??:0
@ 0x7f9fb47aca11 google::LogMessage::~LogMessage() at ??:0
I20251029 16:34:19.906878 8853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:19.907222 8853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:19.907580 8820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3155690d4bb54c97b5a1e7881608ae08. Latest consensus state: current_term: 9 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
@ 0x7f9fcb16eb64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251029 16:34:19.907852 8820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:19.911870 8820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:19.912201 8820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f9fcb16d411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251029 16:34:19.915753 8639 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Sending DeleteTablet for 3 replicas of tablet 17d7be76469746ea80fd24a579212d0a
I20251029 16:34:19.919902 8639 catalog_manager.cc:6719] ReplaceTablet: tablet 17d7be76469746ea80fd24a579212d0a deleted and replaced by tablet 16d35ef1b7da44e3900b509e0a3fd131
I20251029 16:34:19.920296 8639 master_service.cc:954] ReplaceTablet: received request to replace tablet a6d6b65d23ce4b84aab753d124d9e1cf from {username='slave'} at 127.0.0.1:56938
@ 0x7f9fcb17508c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251029 16:34:19.945334 8639 master_service.cc:954] ReplaceTablet: received request to replace tablet a6d6b65d23ce4b84aab753d124d9e1cf from {username='slave'} at 127.0.0.1:56938
I20251029 16:34:19.945875 8639 master_service.cc:954] ReplaceTablet: received request to replace tablet af86f1498654491da54d0da20774edd5 from {username='slave'} at 127.0.0.1:56938
I20251029 16:34:19.946480 8639 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56938:
name: "default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f9fcb174f09 std::_Function_handler<>::_M_invoke() at ??:0
I20251029 16:34:19.948151 7609 tablet_service.cc:1552] Processing DeleteTablet for tablet 17d7be76469746ea80fd24a579212d0a with delete_type TABLET_DATA_DELETED (replaced by tablet 16d35ef1b7da44e3900b509e0a3fd131) from {username='slave'} at 127.0.0.1:47622
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f9fb7520ee9 kudu::Thread::SuperviseThread() at ??:0
I20251029 16:34:19.954699 8842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "3155690d4bb54c97b5a1e7881608ae08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:19.955024 8842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: LEADER
@ 0x7f9fbab2b6db start_thread at ??:0
@ 0x7f9fb365d71f clone at ??:0
I20251029 16:34:19.960565 8866 tablet_replica.cc:333] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46: stopping tablet replica
I20251029 16:34:19.960806 8866 raft_consensus.cc:2243] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 16:34:19.961062 8866 raft_consensus.cc:2272] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:19.961807 8866 ts_tablet_manager.cc:1916] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:19.962060 7751 tablet_service.cc:1552] Processing DeleteTablet for tablet 17d7be76469746ea80fd24a579212d0a with delete_type TABLET_DATA_DELETED (replaced by tablet 16d35ef1b7da44e3900b509e0a3fd131) from {username='slave'} at 127.0.0.1:38996
I20251029 16:34:19.963799 8867 tablet_replica.cc:333] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269: stopping tablet replica
I20251029 16:34:19.964114 8867 raft_consensus.cc:2243] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [term 1 LEADER]: Raft consensus shutting down.
I20251029 16:34:19.964731 8867 raft_consensus.cc:2272] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 16:34:19.965444 8867 ts_tablet_manager.cc:1916] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 16:34:19.969478 8866 ts_tablet_manager.cc:1929] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 16:34:19.969627 8866 log.cc:1199] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-1/wal/wals/17d7be76469746ea80fd24a579212d0a
I20251029 16:34:19.970113 8866 ts_tablet_manager.cc:1950] T 17d7be76469746ea80fd24a579212d0a P 7ec98b5d57094792a1aff2a1df4e7a46: Deleting consensus metadata
I20251029 16:34:19.972882 8867 ts_tablet_manager.cc:1929] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 16:34:19.973037 8867 log.cc:1199] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269: Deleting WAL directory at /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/ts-2/wal/wals/17d7be76469746ea80fd24a579212d0a
I20251029 16:34:19.973521 8867 ts_tablet_manager.cc:1950] T 17d7be76469746ea80fd24a579212d0a P b94ca437df08467bae40554a10450269: Deleting consensus metadata
W20251029 16:34:20.140524 7565 connection.cc:537] server connection from 127.0.0.1:47622 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:20.140552 7705 connection.cc:537] server connection from 127.0.0.1:38996 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:20.140975 7565 connection.cc:537] client connection to 127.6.180.125:44441 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:20.141099 8860 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 16:34:19.918301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.6.180.65:39551)
1029 16:34:19.946132 (+ 27831us) server_negotiation.cc:207] Beginning negotiation
1029 16:34:19.946140 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1029 16:34:19.946163 (+ 23us) server_negotiation.cc:408] Connection header received
1029 16:34:19.946247 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1029 16:34:19.946257 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1029 16:34:19.946366 (+ 109us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1029 16:34:19.950127 (+ 3761us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1029 16:34:19.950450 (+ 323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1029 16:34:19.952156 (+ 1706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1029 16:34:20.140746 (+188590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46892: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":27673,"thread_start_us":33631,"threads_started":1}
W20251029 16:34:20.141135 7414 connection.cc:537] client connection to 127.6.180.125:44441 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:20.141774 7827 connection.cc:537] client connection to 127.6.180.125:44441 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:20.142484 7827 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.6.180.125:44441)
W20251029 16:34:20.142886 7827 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.180.125:44441)
W20251029 16:34:20.143268 7827 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.6.180.125:44441)
W20251029 16:34:20.144132 7827 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.6.180.125:44441)
I20251029 16:34:20.227145 8834 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:20.227301 8834 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: loading cluster ID for follower catalog manager: success
I20251029 16:34:20.231312 8834 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: acquiring CA information for follower catalog manager: success
I20251029 16:34:20.233901 8834 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 16:34:20.346608 7819 heartbeater.cc:499] Master 127.6.180.125:44441 was elected leader, sending a full tablet report...
I20251029 16:34:20.456197 8878 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3155690d4bb54c97b5a1e7881608ae08)
I20251029 16:34:20.456434 8878 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:20.457409 8878 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
W20251029 16:34:20.460124 8766 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
W20251029 16:34:20.460412 8764 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:34:20.460562 8764 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f9bab9976745478595681935778cac; no voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:20.460938 8878 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251029 16:34:21.167259 8884 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3155690d4bb54c97b5a1e7881608ae08)
I20251029 16:34:21.167493 8884 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:21.168224 8884 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
W20251029 16:34:21.170917 8766 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
W20251029 16:34:21.171469 8764 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:34:21.171607 8764 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f9bab9976745478595681935778cac; no voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:21.171986 8884 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251029 16:34:21.591813 6865 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskoTseiL/build/asan/bin/kudu pid 8748
I20251029 16:34:21.601855 8810 generic_service.cc:196] Checking for leaks (request via RPC)
I20251029 16:34:21.711267 6865 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoTseiL/build/asan/bin/kudu with pid 8748
I20251029 16:34:22.360345 8892 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:34:22.360561 8892 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:22.361337 8892 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:34:22.362430 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:34:22.362659 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:34:22.362775 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:34:22.363039 8892 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:34:22.655541 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.124:39271
--webserver_interface=127.6.180.124
--webserver_port=46685
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251029 16:34:23.030754 8893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:34:23.031129 8893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:34:23.031246 8893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:34:23.039901 8893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 16:34:23.040083 8893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:34:23.040232 8893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 16:34:23.040309 8893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 16:34:23.050843 8893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.124:39271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.180.124
--webserver_port=46685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:34:23.052894 8893 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:34:23.054893 8893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 16:34:23.064023 8899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:23.063987 8898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:23.065332 8901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:34:23.065433 8893 server_base.cc:1047] running on GCE node
I20251029 16:34:23.066322 8893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 16:34:23.067873 8893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:34:23.069183 8893 hybrid_clock.cc:648] HybridClock initialized: now 1761755663069136 us; error 29 us; skew 500 ppm
I20251029 16:34:23.069576 8893 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:34:23.071979 8893 webserver.cc:492] Webserver started at http://127.6.180.124:46685/ using document root <none> and password file <none>
I20251029 16:34:23.072875 8893 fs_manager.cc:362] Metadata directory not provided
I20251029 16:34:23.072983 8893 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:34:23.079272 8893 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20251029 16:34:23.082646 8907 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:34:23.084563 8893 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20251029 16:34:23.084776 8893 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
uuid: "29f9bab9976745478595681935778cac"
format_stamp: "Formatted at 2025-10-29 16:33:57 on dist-test-slave-l1bh"
I20251029 16:34:23.085767 8893 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:34:23.105741 8893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 16:34:23.106653 8893 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:34:23.106904 8893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 16:34:23.127956 8958 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.124:39271 every 8 connection(s)
I20251029 16:34:23.127974 8893 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.124:39271
I20251029 16:34:23.129756 8893 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-2/data/info.pb
I20251029 16:34:23.132091 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 8893
I20251029 16:34:23.138828 8959 sys_catalog.cc:263] Verifying existing consensus state
I20251029 16:34:23.143949 8959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Bootstrap starting.
I20251029 16:34:23.158720 8959 log.cc:826] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Log is configured to *not* fsync() on all Append() calls
I20251029 16:34:23.202818 8959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Bootstrap replayed 1/1 log segments. Stats: ops{read=52 overwritten=0 applied=51 ignored=0} inserts{seen=34 ignored=0} mutations{seen=35 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251029 16:34:23.203503 8959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Bootstrap complete.
I20251029 16:34:23.209211 8959 raft_consensus.cc:359] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:23.212329 8959 raft_consensus.cc:740] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f9bab9976745478595681935778cac, State: Initialized, Role: FOLLOWER
I20251029 16:34:23.213091 8959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 51, Last appended: 9.52, Last appended by leader: 52, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:23.213886 8964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:23.214219 8964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:23.215175 8959 sys_catalog.cc:565] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: configured and running, proceeding with master startup.
I20251029 16:34:23.231752 8979 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:23.232005 8979 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: loading cluster ID for follower catalog manager: success
I20251029 16:34:23.236864 8979 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: acquiring CA information for follower catalog manager: success
I20251029 16:34:23.240491 8979 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 16:34:23.240159 8893 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:34:23.246834 8893 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:23.353799 7817 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:34:23.357525 8924 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b94ca437df08467bae40554a10450269" instance_seqno: 1761755639575696) as {username='slave'} at 127.6.180.67:57695; Asking this server to re-register.
I20251029 16:34:23.358549 7817 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:23.360872 8924 ts_manager.cc:194] Registered new tserver with Master: b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:23.530001 7524 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:34:23.531944 8924 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" instance_seqno: 1761755638480543) as {username='slave'} at 127.6.180.65:37151; Asking this server to re-register.
I20251029 16:34:23.532789 7524 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:23.534767 8924 ts_manager.cc:194] Registered new tserver with Master: 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551)
I20251029 16:34:23.536587 7675 heartbeater.cc:344] Connected to a master server at 127.6.180.124:39271
I20251029 16:34:23.538123 8924 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" instance_seqno: 1761755639071323) as {username='slave'} at 127.6.180.66:45537; Asking this server to re-register.
I20251029 16:34:23.538784 7675 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:23.540365 8924 ts_manager.cc:194] Registered new tserver with Master: 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:23.680410 8964 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:23.680684 8964 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:23.682648 8964 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
W20251029 16:34:23.683315 8909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:23.684819 8909 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:23.685057 8911 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:23.685215 8911 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f9bab9976745478595681935778cac; no voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:23.685586 8964 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251029 16:34:24.373585 8987 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:24.373812 8987 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:24.374629 8987 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
W20251029 16:34:24.376920 8911 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
W20251029 16:34:24.377168 8909 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:34:24.377283 8909 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f9bab9976745478595681935778cac; no voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:24.377533 8987 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251029 16:34:25.112171 8994 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:25.112373 8994 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:25.113072 8994 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
W20251029 16:34:25.115290 8911 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
W20251029 16:34:25.115553 8909 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:34:25.115692 8909 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f9bab9976745478595681935778cac; no voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:25.116015 8994 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251029 16:34:26.164436 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.125:44441
--webserver_interface=127.6.180.125
--webserver_port=41231
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251029 16:34:26.288641 9000 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:34:26.288870 9000 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:26.289577 9000 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:34:26.290695 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:34:26.290917 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:34:26.291004 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:34:26.291312 9000 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:34:26.470268 9001 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:26.470511 9001 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:26.472488 9001 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
W20251029 16:34:26.473243 8909 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:26.473511 8911 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441): Network error: Client connection negotiation failed: client connection to 127.6.180.125:44441: connect: Connection refused (error 111)
I20251029 16:34:26.473603 8911 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f9bab9976745478595681935778cac; no voters: 0442da13a653435dba87fc90d8e30d1f, 3155690d4bb54c97b5a1e7881608ae08
I20251029 16:34:26.473846 9001 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251029 16:34:26.524959 8999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:34:26.525338 8999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:34:26.525475 8999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:34:26.534291 8999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 16:34:26.534416 8999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:34:26.534490 8999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 16:34:26.534564 8999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 16:34:26.545650 8999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.125:44441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.180.125
--webserver_port=41231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:34:26.547443 8999 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:34:26.549775 8999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 16:34:26.560180 9009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:26.560657 9008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:26.561578 9011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:34:26.561921 8999 server_base.cc:1047] running on GCE node
I20251029 16:34:26.562883 8999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 16:34:26.564661 8999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:34:26.565899 8999 hybrid_clock.cc:648] HybridClock initialized: now 1761755666565833 us; error 66 us; skew 500 ppm
I20251029 16:34:26.566349 8999 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:34:26.568863 8999 webserver.cc:492] Webserver started at http://127.6.180.125:41231/ using document root <none> and password file <none>
I20251029 16:34:26.569978 8999 fs_manager.cc:362] Metadata directory not provided
I20251029 16:34:26.570118 8999 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:34:26.576347 8999 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20251029 16:34:26.579701 9017 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:34:26.581300 8999 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251029 16:34:26.581501 8999 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
uuid: "3155690d4bb54c97b5a1e7881608ae08"
format_stamp: "Formatted at 2025-10-29 16:33:56 on dist-test-slave-l1bh"
I20251029 16:34:26.582278 8999 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:34:26.601378 8999 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 16:34:26.602348 8999 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:34:26.602689 8999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 16:34:26.623618 8999 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.125:44441
I20251029 16:34:26.623637 9068 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.125:44441 every 8 connection(s)
I20251029 16:34:26.625492 8999 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
I20251029 16:34:26.627764 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 8999
I20251029 16:34:26.634392 9069 sys_catalog.cc:263] Verifying existing consensus state
I20251029 16:34:26.638501 9069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Bootstrap starting.
I20251029 16:34:26.654626 9069 log.cc:826] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Log is configured to *not* fsync() on all Append() calls
I20251029 16:34:26.664817 7526 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:34:26.664841 7677 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:34:26.696593 9069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Bootstrap replayed 1/1 log segments. Stats: ops{read=52 overwritten=0 applied=51 ignored=0} inserts{seen=34 ignored=0} mutations{seen=35 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251029 16:34:26.697291 9069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Bootstrap complete.
I20251029 16:34:26.702965 9069 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 9 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:26.705951 9069 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3155690d4bb54c97b5a1e7881608ae08, State: Initialized, Role: FOLLOWER
I20251029 16:34:26.706701 9069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 51, Last appended: 9.52, Last appended by leader: 52, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:26.707556 9075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:26.707810 9075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:26.708734 9069 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: configured and running, proceeding with master startup.
I20251029 16:34:26.721365 9089 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:26.721493 9089 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: loading cluster ID for follower catalog manager: success
I20251029 16:34:26.726488 9089 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: acquiring CA information for follower catalog manager: success
I20251029 16:34:26.730031 9089 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 16:34:26.731459 8999 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:34:26.737438 8999 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:26.870929 7819 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:34:26.871239 7819 heartbeater.cc:499] Master 127.6.180.125:44441 was elected leader, sending a full tablet report...
I20251029 16:34:26.874742 9033 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b94ca437df08467bae40554a10450269" instance_seqno: 1761755639575696) as {username='slave'} at 127.6.180.67:35695; Asking this server to re-register.
I20251029 16:34:26.875694 7819 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:26.878001 9033 ts_manager.cc:194] Registered new tserver with Master: b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:26.988816 9093 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:26.989022 9093 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:26.989758 9093 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
W20251029 16:34:26.992466 8909 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:34:26.997951 9044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 10 candidate_status { last_received { term: 9 index: 52 } } ignore_live_leader: false dest_uuid: "3155690d4bb54c97b5a1e7881608ae08" is_pre_election: true
I20251029 16:34:26.998584 9044 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f9bab9976745478595681935778cac in term 9.
I20251029 16:34:26.999425 8911 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f
I20251029 16:34:26.999800 9093 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251029 16:34:26.999986 9093 raft_consensus.cc:493] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 16:34:27.000109 9093 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 9 FOLLOWER]: Advancing to term 10
I20251029 16:34:27.005434 9093 raft_consensus.cc:515] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:27.006387 9093 leader_election.cc:290] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 election: Requested vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 3155690d4bb54c97b5a1e7881608ae08 (127.6.180.125:44441)
I20251029 16:34:27.007092 9044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29f9bab9976745478595681935778cac" candidate_term: 10 candidate_status { last_received { term: 9 index: 52 } } ignore_live_leader: false dest_uuid: "3155690d4bb54c97b5a1e7881608ae08"
I20251029 16:34:27.007382 9044 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 9 FOLLOWER]: Advancing to term 10
W20251029 16:34:27.007855 8909 leader_election.cc:336] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:34:27.012482 9044 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f9bab9976745478595681935778cac in term 10.
I20251029 16:34:27.012976 8911 leader_election.cc:304] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f9bab9976745478595681935778cac, 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f
I20251029 16:34:27.013370 9093 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 10 FOLLOWER]: Leader election won for term 10
I20251029 16:34:27.013870 9093 raft_consensus.cc:697] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [term 10 LEADER]: Becoming Leader. State: Replica: 29f9bab9976745478595681935778cac, State: Running, Role: LEADER
I20251029 16:34:27.014400 9093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 9.52, Last appended by leader: 52, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:27.017483 9097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f9bab9976745478595681935778cac. Latest consensus state: current_term: 10 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:27.017827 9097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: LEADER
I20251029 16:34:27.169273 9033 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" instance_seqno: 1761755638480543) as {username='slave'} at 127.6.180.65:50167; Asking this server to re-register.
I20251029 16:34:27.169318 9034 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" instance_seqno: 1761755639071323) as {username='slave'} at 127.6.180.66:59293; Asking this server to re-register.
I20251029 16:34:27.170044 7526 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:27.170117 7677 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:27.171634 9033 ts_manager.cc:194] Registered new tserver with Master: 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:27.171583 9034 ts_manager.cc:194] Registered new tserver with Master: 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551)
W20251029 16:34:27.466087 8909 consensus_peers.cc:597] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac -> Peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Couldn't send request to peer 0442da13a653435dba87fc90d8e30d1f. Status: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251029 16:34:27.522696 9101 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:27.522898 9101 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:27.524768 9101 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:27.525915 9019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:27.527813 9019 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:34:27.532177 8934 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3155690d4bb54c97b5a1e7881608ae08" candidate_term: 11 candidate_status { last_received { term: 9 index: 52 } } ignore_live_leader: false dest_uuid: "29f9bab9976745478595681935778cac" is_pre_election: true
I20251029 16:34:27.533819 9019 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:27.534263 9101 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:34:27.608006 9044 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Refusing update from remote peer 29f9bab9976745478595681935778cac: Log matching property violated. Preceding OpId in replica: term: 9 index: 52. Preceding OpId from leader: term: 10 index: 53. (index mismatch)
I20251029 16:34:27.608887 9097 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 51, Time since last communication: 0.000s
I20251029 16:34:27.612941 9101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f9bab9976745478595681935778cac. Latest consensus state: current_term: 10 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:27.613180 9101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:27.615394 9097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:27.615718 9097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac [sys.catalog]: This master's current role is: LEADER
I20251029 16:34:27.619601 9101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "29f9bab9976745478595681935778cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:27.619897 9101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:27.663877 9099 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251029 16:34:27.666733 9099 catalog_manager.cc:679] Loaded metadata for table default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe]
I20251029 16:34:27.667292 9099 catalog_manager.cc:679] Loaded metadata for table default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab]
I20251029 16:34:27.667635 9099 catalog_manager.cc:679] Loaded metadata for table default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c]
I20251029 16:34:27.668011 9099 catalog_manager.cc:679] Loaded metadata for table default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0]
I20251029 16:34:27.668294 9099 catalog_manager.cc:679] Loaded metadata for table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e]
I20251029 16:34:27.668596 9099 catalog_manager.cc:679] Loaded metadata for table default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b]
I20251029 16:34:27.668844 9099 catalog_manager.cc:679] Loaded metadata for table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f]
I20251029 16:34:27.669104 9099 catalog_manager.cc:679] Loaded metadata for table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80]
I20251029 16:34:27.669363 9099 catalog_manager.cc:679] Loaded metadata for table default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6]
I20251029 16:34:27.671196 9099 tablet_loader.cc:96] loaded metadata for tablet 167c0aa33db046408bbc9bcd0844d152 (table default.table_4d4ea90e32c7496a91ee99d2ca5a1f01 [id=d31229edb9834ddab88752bc8d367e80])
I20251029 16:34:27.671660 9099 tablet_loader.cc:96] loaded metadata for tablet 16d35ef1b7da44e3900b509e0a3fd131 (table default.table_6129e3888a3346669ab7c8982593f438 [id=54c21e8f41bb4f03be83c5cce599242c])
I20251029 16:34:27.672433 9099 tablet_loader.cc:96] loaded metadata for tablet 28e1452396924512ac30d309085628ee (table default.table_dfc244c581534b678f83267a7d10b58c [id=c3822131c9e8475eb4d6bb49bf60e48f])
I20251029 16:34:27.673375 9099 tablet_loader.cc:96] loaded metadata for tablet 48947ffd82bf4768bed14e9bea56a60a (table default.table_e6381c7d53654326b15714d30d7eba93 [id=370cf332e83748ae807762a392d183ab])
I20251029 16:34:27.673732 9099 tablet_loader.cc:96] loaded metadata for tablet 4af807ac4f78451592a0a6788af12044 (table default.table_809b2f55b3594aa1aab4872f17b07c4e [id=de52525dd5684e4da86d8f8c77ad80d6])
I20251029 16:34:27.674150 9099 tablet_loader.cc:96] loaded metadata for tablet 4f5e322159e6488d98400bfb9249522d (table default.table_571d6fa0fe7e4782b8b0c692012fbd81 [id=a3cb6d437da645b488adaae8b577b07e])
I20251029 16:34:27.674798 9099 tablet_loader.cc:96] loaded metadata for tablet 8197c292e24348e1b633ba29e22a0bec (table default.table_789f901f60d34caf9ddd2eb0e44c9dd2 [id=6e4eb1073fdf46d19d4e3ae42a186ce0])
I20251029 16:34:27.675889 9099 tablet_loader.cc:96] loaded metadata for tablet acbbc2208b744b6c8ce5ea0fe699d29e (table default.table_0982a1f0341d4f3f9e7f27ed71278b4d [id=1f98c7c5ebc94e9fa5111bc7845d9abe])
I20251029 16:34:27.676733 9099 tablet_loader.cc:96] loaded metadata for tablet f5e3fd53975145e9acbc9dff7a2b0cb9 (table default.table_207ddcecc1c54b149e04a4d6f8190acb [id=bac0388305c24aa196e9a4e444e6915b])
I20251029 16:34:27.677037 9099 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251029 16:34:27.678251 9099 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:27.678344 9099 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251029 16:34:27.679754 9099 catalog_manager.cc:1514] Loading token signing keys...
I20251029 16:34:27.680785 9099 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 29f9bab9976745478595681935778cac: Loaded TSK: 0
I20251029 16:34:27.681748 9099 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251029 16:34:27.682790 9099 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251029 16:34:27.683737 9099 catalog_manager.cc:5871] Last processed Hive Metastore notification event ID: -1
I20251029 16:34:27.735931 8924 master_service.cc:954] ReplaceTablet: received request to replace tablet 82b72b5cd6a1496eb09ac78395a285a1 from {username='slave'} at 127.0.0.1:54136
I20251029 16:34:27.736096 8923 master_service.cc:954] ReplaceTablet: received request to replace tablet 4233e0df604c4ff8a7e60528788d7d07 from {username='slave'} at 127.0.0.1:54136
I20251029 16:34:27.737069 8919 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54136:
table { table_name: "default.Table_809b2f55b3594aa1aab4872f17b07c4e" } new_table_name: "default.Table_feb27132a62f47388739326e4fb4118c" modify_external_catalogs: true
I20251029 16:34:27.736996 8921 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54136:
name: "default.Table_70f60abb54374bb5a1a1475de5feec56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:27.738420 8923 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54136:
name: "default.Table_4bedb57357b24f78a75d1f362473ceff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:27.738122 8924 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54136:
name: "default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:27.738718 8916 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54136:
name: "default.Table_3a47f0a3057e46acaf74a5e33bffb30a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:27.739332 8917 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54136:
table { table_name: "default.Table_dfc244c581534b678f83267a7d10b58c" } new_table_name: "default.Table_35daf50c962249fd983f0e49e1f7960a" modify_external_catalogs: true
I20251029 16:34:27.739904 8920 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54136:
name: "default.Table_ca14635728c14a7fbd4bce9c90ec208d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 16:34:27.740794 8915 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54136:
name: "default.Table_e4dfb9e8508e41669fcd08f8c79b24f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251029 16:34:27.743396 8978 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1761755639, eventType=CREATE_TABLE, dbName=default, tableName=table_b513b87e8a1943eea3e24140ede4921d, message={"server":"","servicePrincipal":"","db":"default","table":"table_b513b87e8a1943eea3e24140ede4921d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b513b87e8a1943eea3e24140ede4921d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761755639},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/warehouse/table_b513b87e8a1943eea3e24140ede4921d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b513b87e8a1943eea3e24140ede4921d\",\"kudu.table_id\":\"1398ac3c2f22427b96af2e9ff15e62d4\",\"transient_lastDdlTime\":\"1761755639\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"415c0df9d5524c9a82ceee0f02442bac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761755640,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1761755640, eventType=CREATE_TABLE, dbName=default, tableName=table_4d4ea90e32c7496a91ee99d2ca5a1f01, message={"server":"","servicePrincipal":"","db":"default","table":"table_4d4ea90e32c7496a91ee99d2ca5a1f01","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_4d4ea90e32c7496a91ee99d2ca5a1f01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1761755640},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/warehouse/table_4d4ea90e32c7496a91ee99d2ca5a1f01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_4d4ea90e32c7496a91ee99d2ca5a1f01\",\"kudu.table_id\":\"d31229edb9834ddab88752bc8d367e80\",\"transient_lastDdlTime\":\"1761755640\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"415c0df9d5524c9a82ceee0f02442bac\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1761755640,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251029 16:34:27.744297 8918 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54136:
table { table_name: "default.Table_6129e3888a3346669ab7c8982593f438" } new_table_name: "default.Table_323e8fdbeff34a12add0ad9e4c97accd" modify_external_catalogs: true
@ 0x7f089932fdcd google::LogMessage::Fail() at ??:0
I20251029 16:34:27.746799 8922 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54136:
name: "default.Table_700613fee8534a63b231420a5e166fb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f0899333b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f089932f7cc google::LogMessage::Flush() at ??:0
@ 0x7f089932fa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f08afcf1b64 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f08afcf0411 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f08afcf808c kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f08afcf7f09 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f089c0a3ee9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f089f6ae6db start_thread at ??:0
@ 0x7f08981e071f clone at ??:0
W20251029 16:34:27.960353 7825 connection.cc:537] client connection to 127.6.180.124:39271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:27.960893 7705 connection.cc:537] client connection to 127.6.180.124:39271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 16:34:27.961081 7825 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.6.180.124:39271)
W20251029 16:34:27.961644 7825 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.180.124:39271)
W20251029 16:34:27.962023 7825 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.6.180.124:39271)
W20251029 16:34:27.962348 7825 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.6.180.124:39271)
I20251029 16:34:28.115229 9101 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29f9bab9976745478595681935778cac)
I20251029 16:34:28.115484 9101 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:28.116179 9101 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:28.118486 9019 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:28.118698 9019 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:28.118803 9019 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:28.119100 9101 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:34:28.839968 6865 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskoTseiL/build/asan/bin/kudu pid 8999
I20251029 16:34:28.847343 9064 generic_service.cc:196] Checking for leaks (request via RPC)
I20251029 16:34:28.957892 9137 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 29f9bab9976745478595681935778cac)
I20251029 16:34:28.958257 6865 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoTseiL/build/asan/bin/kudu with pid 8999
I20251029 16:34:28.958250 9137 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:29.810320 6865 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoTseiL/build/asan/bin/kudu
/tmp/dist-test-taskoTseiL/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.125:44441
--webserver_interface=127.6.180.125
--webserver_port=41231
--builtin_ntp_servers=127.6.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251029 16:34:30.182360 9139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 16:34:30.182781 9139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 16:34:30.182912 9139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 16:34:30.191810 9139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 16:34:30.191993 9139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 16:34:30.192101 9139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 16:34:30.192173 9139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 16:34:30.202737 9139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:33357
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.180.125:44441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.180.125
--webserver_port=41231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 16:25:22 UTC on 24a791456cd2
build id 8776
ASAN enabled
I20251029 16:34:30.204577 9139 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 16:34:30.206523 9139 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 16:34:30.215580 9145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:30.215567 9144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 16:34:30.216717 9147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 16:34:30.217674 9139 server_base.cc:1047] running on GCE node
I20251029 16:34:30.218518 9139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 16:34:30.220171 9139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 16:34:30.221349 9139 hybrid_clock.cc:648] HybridClock initialized: now 1761755670221297 us; error 46 us; skew 500 ppm
I20251029 16:34:30.221776 9139 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 16:34:30.224211 9139 webserver.cc:492] Webserver started at http://127.6.180.125:41231/ using document root <none> and password file <none>
I20251029 16:34:30.224994 9139 fs_manager.cc:362] Metadata directory not provided
I20251029 16:34:30.225100 9139 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 16:34:30.231405 9139 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20251029 16:34:30.234709 9153 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 16:34:30.236487 9139 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20251029 16:34:30.236682 9139 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data,/tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
uuid: "3155690d4bb54c97b5a1e7881608ae08"
format_stamp: "Formatted at 2025-10-29 16:33:56 on dist-test-slave-l1bh"
I20251029 16:34:30.237491 9139 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 16:34:30.253968 9139 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 16:34:30.254940 9139 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 16:34:30.255293 9139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 16:34:30.276750 9139 rpc_server.cc:307] RPC server started. Bound to: 127.6.180.125:44441
I20251029 16:34:30.276769 9206 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.180.125:44441 every 8 connection(s)
I20251029 16:34:30.278602 9139 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskoTseiL/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1761755621690668-6865-0/minicluster-data/master-1/data/info.pb
I20251029 16:34:30.284543 6865 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoTseiL/build/asan/bin/kudu as pid 9139
I20251029 16:34:30.288170 9207 sys_catalog.cc:263] Verifying existing consensus state
I20251029 16:34:30.295782 9207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Bootstrap starting.
I20251029 16:34:30.314891 9207 log.cc:826] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Log is configured to *not* fsync() on all Append() calls
I20251029 16:34:30.355715 9207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Bootstrap replayed 1/1 log segments. Stats: ops{read=53 overwritten=0 applied=53 ignored=0} inserts{seen=34 ignored=0} mutations{seen=38 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251029 16:34:30.356540 9207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: Bootstrap complete.
I20251029 16:34:30.362594 9207 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:30.363399 9207 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3155690d4bb54c97b5a1e7881608ae08, State: Initialized, Role: FOLLOWER
I20251029 16:34:30.364241 9207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 53, Last appended: 10.53, Last appended by leader: 53, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:30.365145 9212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } } }
I20251029 16:34:30.365417 9212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 16:34:30.366232 9207 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [sys.catalog]: configured and running, proceeding with master startup.
I20251029 16:34:30.381747 9227 catalog_manager.cc:1269] Loaded cluster ID: 415c0df9d5524c9a82ceee0f02442bac
I20251029 16:34:30.381974 9227 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: loading cluster ID for follower catalog manager: success
I20251029 16:34:30.386994 9227 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: acquiring CA information for follower catalog manager: success
I20251029 16:34:30.390565 9227 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 16:34:30.393332 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:34:30.399462 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:30.488735 7819 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:34:30.491510 9171 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b94ca437df08467bae40554a10450269" instance_seqno: 1761755639575696) as {username='slave'} at 127.6.180.67:49603; Asking this server to re-register.
I20251029 16:34:30.492372 7819 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:30.494678 9171 ts_manager.cc:194] Registered new tserver with Master: b94ca437df08467bae40554a10450269 (127.6.180.67:38629)
I20251029 16:34:30.692570 7526 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:34:30.692965 7677 heartbeater.cc:344] Connected to a master server at 127.6.180.125:44441
I20251029 16:34:30.694248 9171 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" instance_seqno: 1761755638480543) as {username='slave'} at 127.6.180.65:42199; Asking this server to re-register.
I20251029 16:34:30.694589 9170 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" instance_seqno: 1761755639071323) as {username='slave'} at 127.6.180.66:50301; Asking this server to re-register.
I20251029 16:34:30.694901 7526 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:30.695235 7677 heartbeater.cc:461] Registering TS with master...
I20251029 16:34:30.696550 9170 ts_manager.cc:194] Registered new tserver with Master: 4a7cbc2353a042bba0a6103565a2828c (127.6.180.65:39551)
I20251029 16:34:30.696976 9171 ts_manager.cc:194] Registered new tserver with Master: 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
I20251029 16:34:30.986248 9231 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:30.986524 9231 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:30.988364 9231 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:30.989439 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:30.990844 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:30.991080 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:30.991192 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:30.991626 9231 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:34:31.311437 9100 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:34:31.311657 9100 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:31.312381 9100 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:34:31.313493 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:34:31.313683 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:34:31.313786 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:34:31.314023 9100 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:34:31.686555 9234 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:31.686888 9234 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:31.687706 9234 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:31.689750 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:31.689939 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:31.690037 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:31.690296 9234 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:34:32.653187 6865 master-stress-test.cc:577] Killing test thread 1/25
I20251029 16:34:32.773550 9239 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:32.773752 9239 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:32.774431 9239 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:32.776983 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:32.777230 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:32.777387 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:32.777930 9239 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:34:33.800269 9244 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:33.800484 9244 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:33.801126 9244 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:33.803601 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:33.803830 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:33.803972 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:33.804250 9244 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:34:35.401448 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:34:35.401690 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:35.470351 9251 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:35.470573 9251 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:35.471246 9251 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:35.477370 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:35.477617 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:35.477722 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:35.478011 9251 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:34:37.697196 9255 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:37.697454 9255 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:37.698215 9255 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:37.699401 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20251029 16:34:37.700891 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:37.701115 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:37.701202 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:37.701445 9255 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:34:39.697017 9262 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:39.697283 9262 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:39.698050 9262 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:39.700376 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:39.700553 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:39.700659 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:39.700907 9262 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:34:40.403317 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:34:40.403517 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:40.588124 9100 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:34:40.588354 9100 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:40.589196 9100 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:34:40.590224 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:34:40.590435 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:34:40.590504 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:34:40.590742 9100 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:34:42.523628 9267 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:42.523856 9267 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:42.524660 9267 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:42.527323 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:42.527555 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:42.527655 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:42.528008 9267 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:34:44.735769 9274 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:44.736063 9274 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:44.736770 9274 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:44.738161 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20251029 16:34:44.741384 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:44.741679 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:44.741771 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:44.742056 9274 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:34:45.405154 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:34:45.405356 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:49.961756 9272 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:34:49.961966 9272 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:49.962718 9272 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:34:49.963835 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:34:49.964103 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:34:49.964243 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:34:49.964525 9272 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:34:50.406886 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:34:50.407130 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:55.408803 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:34:55.409019 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:34:57.518849 9298 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:34:57.519093 9298 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:34:57.519878 9298 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:34:57.521211 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:34:57.522511 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:34:57.522735 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:34:57.522863 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:34:57.523198 9298 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:34:59.606981 9303 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:34:59.607251 9303 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:34:59.608134 9303 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:34:59.609153 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:34:59.609411 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:34:59.609535 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:34:59.609860 9303 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251029 16:35:00.121598 7525 heartbeater.cc:646] Failed to heartbeat to 127.6.180.126:36819 (86 consecutive failures): Network error: Failed to ping master at 127.6.180.126:36819: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:35:00.121647 7676 heartbeater.cc:646] Failed to heartbeat to 127.6.180.126:36819 (86 consecutive failures): Network error: Failed to ping master at 127.6.180.126:36819: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:35:00.410612 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:35:00.410820 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251029 16:35:00.433099 7818 heartbeater.cc:646] Failed to heartbeat to 127.6.180.126:36819 (87 consecutive failures): Network error: Failed to ping master at 127.6.180.126:36819: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:35:05.412494 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:35:05.412755 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:35:10.414470 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:35:10.414683 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:35:11.388643 9317 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:35:11.388850 9317 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:35:11.389536 9317 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:35:11.391045 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:35:11.393139 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:35:11.393371 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:35:11.393463 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:35:11.393754 9317 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:35:14.095399 9315 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:35:14.095650 9315 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:35:14.096521 9315 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:35:14.097609 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:35:14.097818 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:35:14.097916 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:35:14.098230 9315 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:35:15.416406 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:35:15.416631 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:35:17.644243 9315 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:35:17.644492 9315 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:35:17.645424 9315 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:35:17.646464 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:35:17.646682 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:35:17.646778 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:35:17.647089 9315 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:35:20.418287 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:35:20.418515 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:35:24.386116 9329 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:35:24.386394 9329 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:35:24.387112 9329 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:35:24.388427 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:35:24.389736 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:35:24.389918 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:35:24.389997 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:35:24.390276 9329 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:35:25.420048 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:35:25.420269 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:35:30.421788 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:35:30.422001 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:35:31.860091 9341 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:35:31.860339 9341 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:35:31.861014 9341 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:35:31.862360 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:35:31.865276 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:35:31.865540 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:35:31.865628 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:35:31.865928 9341 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:35:35.220713 9315 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:35:35.220916 9315 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:35:35.221793 9315 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:35:35.222745 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:35:35.223013 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:35:35.223125 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:35:35.223400 9315 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:35:35.423484 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:35:35.423689 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:35:40.425239 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:35:40.425468 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:35:45.427199 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:35:45.427433 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:35:48.331068 9365 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:35:48.331274 9365 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:35:48.332232 9365 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:35:48.333729 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:35:48.335024 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:35:48.335264 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:35:48.335361 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:35:48.335644 9365 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:35:50.429011 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:35:50.429230 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:35:53.960100 9370 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:35:53.960335 9370 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:35:53.961249 9370 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:35:53.962301 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:35:53.962535 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:35:53.962630 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:35:53.962925 9370 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:35:55.430871 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:35:55.431067 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251029 16:36:00.279320 7525 heartbeater.cc:646] Failed to heartbeat to 127.6.180.126:36819 (206 consecutive failures): Network error: Failed to ping master at 127.6.180.126:36819: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:36:00.279371 7676 heartbeater.cc:646] Failed to heartbeat to 127.6.180.126:36819 (206 consecutive failures): Network error: Failed to ping master at 127.6.180.126:36819: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:36:00.432610 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:36:00.432797 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251029 16:36:00.586944 7818 heartbeater.cc:646] Failed to heartbeat to 127.6.180.126:36819 (207 consecutive failures): Network error: Failed to ping master at 127.6.180.126:36819: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:36:02.394151 9393 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:36:02.394385 9393 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:36:02.395072 9393 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:36:02.396477 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:36:02.397933 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:36:02.398180 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:36:02.398284 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:36:02.398577 9393 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:36:04.629155 9401 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:36:04.629390 9401 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:36:04.630087 9401 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:36:04.633558 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:36:04.633747 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:36:04.633841 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:36:04.634151 9401 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:36:05.267028 9370 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:36:05.267267 9370 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:36:05.267971 9370 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:36:05.269135 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:36:05.269354 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:36:05.269474 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:36:05.269733 9370 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:36:05.434734 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:36:05.435012 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:36:10.436959 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:36:10.437218 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:36:12.406401 9370 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:36:12.406658 9370 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:36:12.407493 9370 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:36:12.408531 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:36:12.409049 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:36:12.409149 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:36:12.409464 9370 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:36:12.544543 9413 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:36:12.544780 9413 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:36:12.545511 9413 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:36:12.546886 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251029 16:36:12.549115 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:36:12.549374 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:36:12.549499 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:36:12.549810 9413 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:36:15.438836 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:36:15.439071 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:36:20.440836 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:36:20.441069 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:36:20.966259 9425 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:36:20.966477 9425 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:36:20.967310 9425 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:36:20.968641 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:36:20.971280 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
W20251029 16:36:20.971467 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:36:20.971576 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:36:20.971931 9425 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:36:25.442622 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:36:25.442855 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:36:25.911630 9418 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:36:25.911847 9418 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:36:25.912598 9418 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:36:25.913669 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:36:25.913851 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:36:25.913959 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:36:25.914256 9418 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:36:30.449295 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:36:30.449512 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:36:30.457233 9159 maintenance_manager.cc:419] P 3155690d4bb54c97b5a1e7881608ae08: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033359
I20251029 16:36:30.506840 9158 maintenance_manager.cc:643] P 3155690d4bb54c97b5a1e7881608ae08: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.049s user 0.026s sys 0.004s Metrics: {"bytes_written":15653,"cfile_init":1,"dirs.queue_time_us":521,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":886,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":659,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":34,"thread_start_us":626,"threads_started":2,"wal-append.queue_time_us":395}
I20251029 16:36:30.508361 9159 maintenance_manager.cc:419] P 3155690d4bb54c97b5a1e7881608ae08: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1692 bytes on disk
I20251029 16:36:30.509160 9158 maintenance_manager.cc:643] P 3155690d4bb54c97b5a1e7881608ae08: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20251029 16:36:34.517480 9449 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:36:34.517700 9449 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:36:34.518355 9449 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:36:34.520644 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:36:34.522364 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:36:34.522581 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:36:34.522679 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:36:34.522948 9449 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:36:35.450942 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:36:35.451146 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:36:37.553925 9418 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:36:37.554164 9418 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:36:37.554976 9418 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:36:37.556002 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:36:37.556236 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:36:37.556352 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:36:37.556655 9418 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:36:40.452862 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:36:40.453100 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:36:45.357949 9504 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:36:45.358182 9504 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:36:45.358896 9504 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:36:45.360190 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:36:45.366775 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:36:45.367038 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:36:45.367151 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:36:45.367455 9504 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:36:45.454444 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:36:45.454660 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:36:50.456411 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:36:50.456631 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:36:50.897198 9538 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:36:50.897436 9538 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:36:50.898129 9538 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:36:50.899194 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:36:50.899391 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:36:50.899502 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:36:50.899781 9538 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:36:55.458281 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:36:55.458490 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:36:59.012603 9544 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:36:59.012831 9544 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:36:59.013669 9544 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:36:59.014598 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:36:59.014844 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:36:59.014927 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:36:59.015205 9544 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251029 16:37:00.431496 7525 heartbeater.cc:646] Failed to heartbeat to 127.6.180.126:36819 (326 consecutive failures): Network error: Failed to ping master at 127.6.180.126:36819: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:37:00.431545 7676 heartbeater.cc:646] Failed to heartbeat to 127.6.180.126:36819 (326 consecutive failures): Network error: Failed to ping master at 127.6.180.126:36819: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:37:00.460726 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:37:00.460954 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251029 16:37:00.744676 7818 heartbeater.cc:646] Failed to heartbeat to 127.6.180.126:36819 (327 consecutive failures): Network error: Failed to ping master at 127.6.180.126:36819: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:37:00.797288 9586 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:37:00.797503 9586 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:37:00.798313 9586 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:37:00.798913 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:37:00.800246 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:37:00.800407 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:37:00.800526 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:37:00.800789 9586 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:37:05.462602 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:37:05.462858 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:37:10.464752 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:37:10.465036 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:37:10.465188 9635 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:37:10.465373 9635 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:37:10.465988 9635 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:37:10.466538 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:37:10.468696 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:37:10.468911 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:37:10.469033 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:37:10.469621 9635 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:37:15.467025 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:37:15.467264 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:37:17.916126 9631 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:37:17.916357 9631 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:37:17.917205 9631 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:37:17.918320 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:37:17.918521 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:37:17.918604 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:37:17.918835 9631 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:37:20.468753 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:37:20.468959 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:37:25.470387 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:37:25.470592 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:37:26.013091 9702 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:37:26.013303 9702 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:37:26.014180 9702 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:37:26.015342 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:37:26.016741 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:37:26.016909 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:37:26.017014 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:37:26.017253 9702 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:37:28.848501 9719 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:37:28.848748 9719 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:37:28.849440 9719 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:37:28.850354 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:37:28.850567 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:37:28.850667 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:37:28.850955 9719 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:37:30.472258 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:37:30.472481 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:37:35.474051 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:37:35.474269 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:37:40.475982 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:37:40.476207 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:37:41.916543 9779 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:37:41.916780 9779 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:37:41.917574 9779 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:37:41.918779 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:37:41.920118 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:37:41.920322 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:37:41.920431 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:37:41.920692 9779 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:37:43.433230 9778 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:37:43.433424 9778 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:37:43.434396 9778 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:37:43.435195 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:37:43.435405 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:37:43.435514 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:37:43.435791 9778 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:37:45.477680 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:37:45.477881 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:37:50.479583 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:37:50.479792 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:37:55.002615 9837 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:37:55.002817 9837 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:37:55.003415 9837 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:37:55.004611 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:37:55.005983 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:37:55.006183 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:37:55.006289 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:37:55.006521 9837 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:37:55.480799 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:37:55.480985 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:38:00.279026 9859 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:38:00.279250 9859 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:38:00.280184 9859 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:38:00.281243 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:38:00.281454 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:38:00.281536 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:38:00.281841 9859 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:38:00.482510 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:38:00.482719 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251029 16:38:00.583616 7525 heartbeater.cc:646] Failed to heartbeat to 127.6.180.126:36819 (446 consecutive failures): Network error: Failed to ping master at 127.6.180.126:36819: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:38:00.584051 7676 heartbeater.cc:646] Failed to heartbeat to 127.6.180.126:36819 (446 consecutive failures): Network error: Failed to ping master at 127.6.180.126:36819: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:38:00.900759 7818 heartbeater.cc:646] Failed to heartbeat to 127.6.180.126:36819 (447 consecutive failures): Network error: Failed to ping master at 127.6.180.126:36819: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:38:02.222113 9870 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:38:02.222345 9870 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:38:02.223124 9870 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:38:02.224326 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:38:02.225961 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:38:02.226157 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:38:02.226279 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:38:02.226538 9870 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:38:05.484459 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:38:05.484663 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:38:10.486389 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:38:10.486588 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:38:11.269752 9898 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:38:11.269975 9898 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:38:11.270594 9898 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:38:11.271649 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:38:11.271862 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:38:11.271998 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:38:11.272281 9898 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:38:11.854110 9908 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:38:11.854331 9908 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:38:11.855041 9908 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:38:11.856271 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:38:11.861351 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:38:11.861613 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:38:11.861722 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:38:11.862001 9908 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:38:15.488272 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:38:15.488487 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:38:17.439391 9918 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:38:17.439620 9918 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:38:17.440544 9918 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:38:17.441512 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:38:17.441745 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:38:17.441833 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:38:17.442114 9918 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:38:20.490191 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:38:20.490419 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:38:22.856086 9994 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:38:22.856360 9994 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:38:22.857136 9994 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:38:22.858423 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:38:22.859742 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:38:22.859985 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:38:22.860091 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:38:22.860327 9994 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:38:24.388175 10015 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:38:24.388377 10015 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:38:24.389015 10015 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:38:24.390126 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:38:24.390341 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:38:24.390451 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:38:24.390717 10015 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:38:25.491714 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:38:25.491890 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:38:30.493461 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:38:30.493659 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:38:31.023675 10088 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:38:31.023855 10088 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:38:31.024538 10088 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:38:31.025662 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:38:31.026939 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:38:31.027136 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:38:31.027243 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:38:31.027514 10088 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:38:34.794128 10095 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:38:34.794323 10095 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:38:34.795172 10095 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:38:34.796213 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:38:34.796411 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:38:34.796496 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:38:34.796744 10095 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:38:35.495091 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:38:35.495311 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:38:36.417464 10148 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:38:36.417675 10148 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:38:36.418341 10148 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:38:36.419355 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:38:36.420518 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:38:36.420696 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:38:36.420790 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:38:36.421029 10148 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:38:40.496939 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:38:40.497119 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:38:42.172123 10138 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:38:42.172358 10138 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:38:42.173166 10138 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:38:42.174285 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:38:42.174530 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:38:42.174674 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:38:42.174993 10138 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:38:45.499040 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:38:45.499287 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:38:50.501291 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:38:50.501500 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:38:50.928409 10297 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:38:50.928714 10297 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:38:50.929662 10297 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:38:50.931211 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:38:50.932842 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:38:50.933091 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:38:50.933238 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:38:50.933543 10297 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:38:55.503515 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:38:55.503744 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:38:59.693482 10368 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:38:59.693698 10368 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:38:59.695004 10368 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:38:59.696048 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:38:59.696269 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:38:59.696358 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:38:59.696653 10368 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:39:00.505442 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:39:00.505695 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251029 16:39:00.798419 7524 heartbeater.cc:646] Failed to heartbeat to 127.6.180.124:39271 (544 consecutive failures): Network error: Failed to ping master at 127.6.180.124:39271: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
W20251029 16:39:00.798565 7676 heartbeater.cc:646] Failed to heartbeat to 127.6.180.126:36819 (566 consecutive failures): Network error: Failed to ping master at 127.6.180.126:36819: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:39:01.055640 7818 heartbeater.cc:646] Failed to heartbeat to 127.6.180.126:36819 (567 consecutive failures): Network error: Failed to ping master at 127.6.180.126:36819: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
I20251029 16:39:01.720894 10403 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:39:01.721205 10403 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:39:01.722138 10403 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:39:01.723410 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:39:01.725085 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:39:01.725260 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:39:01.725404 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:39:01.725669 10403 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:39:05.507443 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:39:05.507650 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:39:09.070173 10476 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:39:09.070487 10476 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:39:09.071405 10476 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:39:09.072575 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251029 16:39:09.074554 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:39:09.074755 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:39:09.074882 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:39:09.075168 10476 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:39:10.509200 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:39:10.509400 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:39:10.607746 10495 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:39:10.608011 10495 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:39:10.608693 10495 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:39:10.610670 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:39:10.610832 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:39:10.610911 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:39:10.611178 10495 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:39:15.510977 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:39:15.511195 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:39:15.951210 10545 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 16:39:15.951421 10545 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442da13a653435dba87fc90d8e30d1f" member_type: VOTER last_known_addr { host: "127.6.180.126" port: 36819 } } peers { permanent_uuid: "3155690d4bb54c97b5a1e7881608ae08" member_type: VOTER last_known_addr { host: "127.6.180.125" port: 44441 } } peers { permanent_uuid: "29f9bab9976745478595681935778cac" member_type: VOTER last_known_addr { host: "127.6.180.124" port: 39271 } }
I20251029 16:39:15.952142 10545 leader_election.cc:290] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819), 29f9bab9976745478595681935778cac (127.6.180.124:39271)
W20251029 16:39:15.952761 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251029 16:39:15.953982 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0442da13a653435dba87fc90d8e30d1f (127.6.180.126:36819): Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111)
W20251029 16:39:15.954180 9155 leader_election.cc:336] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 29f9bab9976745478595681935778cac (127.6.180.124:39271): Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
I20251029 16:39:15.954272 9155 leader_election.cc:304] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3155690d4bb54c97b5a1e7881608ae08; no voters: 0442da13a653435dba87fc90d8e30d1f, 29f9bab9976745478595681935778cac
I20251029 16:39:15.954563 10545 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3155690d4bb54c97b5a1e7881608ae08 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251029 16:39:16.769505 10551 raft_consensus.cc:493] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b94ca437df08467bae40554a10450269)
I20251029 16:39:16.769711 10551 raft_consensus.cc:515] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a7cbc2353a042bba0a6103565a2828c" member_type: VOTER last_known_addr { host: "127.6.180.65" port: 39551 } } peers { permanent_uuid: "b94ca437df08467bae40554a10450269" member_type: VOTER last_known_addr { host: "127.6.180.67" port: 38629 } } peers { permanent_uuid: "7ec98b5d57094792a1aff2a1df4e7a46" member_type: VOTER last_known_addr { host: "127.6.180.66" port: 33379 } }
I20251029 16:39:16.770632 10551 leader_election.cc:290] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b94ca437df08467bae40554a10450269 (127.6.180.67:38629), 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379)
W20251029 16:39:16.771641 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b94ca437df08467bae40554a10450269 (127.6.180.67:38629): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
W20251029 16:39:16.771871 7410 leader_election.cc:343] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ec98b5d57094792a1aff2a1df4e7a46 (127.6.180.66:33379): Not found: Tablet not found: 17d7be76469746ea80fd24a579212d0a
I20251029 16:39:16.772024 7410 leader_election.cc:304] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a7cbc2353a042bba0a6103565a2828c; no voters: 7ec98b5d57094792a1aff2a1df4e7a46, b94ca437df08467bae40554a10450269
I20251029 16:39:16.772298 10551 raft_consensus.cc:2749] T 17d7be76469746ea80fd24a579212d0a P 4a7cbc2353a042bba0a6103565a2828c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 16:39:20.043558 9159 maintenance_manager.cc:419] P 3155690d4bb54c97b5a1e7881608ae08: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1692 bytes on disk
I20251029 16:39:20.049515 9158 maintenance_manager.cc:643] P 3155690d4bb54c97b5a1e7881608ae08: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.005s sys 0.000s Metrics: {"dirs.queue_time_us":453,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":115,"thread_start_us":574,"threads_started":2,"wal-append.queue_time_us":374}
I20251029 16:39:20.512908 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:39:20.513114 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 16:39:25.514916 9139 master_runner.cc:186] Error getting master registration for 127.6.180.126:36819: Network error: Client connection negotiation failed: client connection to 127.6.180.126:36819: connect: Connection refused (error 111),
I20251029 16:39:25.515165 9139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251029 16:39:28.449016 7825 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
W20251029 16:39:28.449393 7825 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
W20251029 16:39:28.449508 7825 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
W20251029 16:39:28.449632 7825 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 *** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1761755968 (unix time) try "date -d @1761755968" if you are using GNU date ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460912 7847 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463057 7841 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463057 7841 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463109 7835 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cf493c1cc4f44b069dba62ecf1ad3676 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463057 7841 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463109 7835 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cf493c1cc4f44b069dba62ecf1ad3676 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463188 7836 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3a47f0a3057e46acaf74a5e33bffb30a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463057 7841 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463109 7835 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cf493c1cc4f44b069dba62ecf1ad3676 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463188 7836 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3a47f0a3057e46acaf74a5e33bffb30a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465879 7840 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70f60abb54374bb5a1a1475de5feec56 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465878 *** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463057 7841 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463109 7835 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cf493c1cc4f44b069dba62ecf1ad3676 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463188 7836 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3a47f0a3057e46acaf74a5e33bffb30a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465879 7840 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70f60abb54374bb5a1a1475de5feec56 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465878 7852 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463057 7841 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463109 7835 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cf493c1cc4f44b069dba62ecf1ad3676 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463188 7836 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3a47f0a3057e46acaf74a5e33bffb30a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465879 7840 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70f60abb54374bb5a1a1475de5feec56 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465878 7852 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467108 7845 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463057 7841 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463109 7835 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cf493c1cc4f44b069dba62ecf1ad3676 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463188 7836 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3a47f0a3057e46acaf74a5e33bffb30a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465879 7840 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70f60abb54374bb5a1a1475de5feec56 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465878 7852 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467108 7845 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467270 7834 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_700613fee8534a63b231420a5e166fb5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463057 7841 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463109 7835 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cf493c1cc4f44b069dba62ecf1ad3676 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463188 7836 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3a47f0a3057e46acaf74a5e33bffb30a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465879 7840 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70f60abb54374bb5a1a1475de5feec56 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465878 7852 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467108 7845 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467270 7834 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_700613fee8534a63b231420a5e166fb5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467375 7838 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4bedb57357b24f78a75d1f362473ceff on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463057 7841 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463109 7835 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cf493c1cc4f44b069dba62ecf1ad3676 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463188 7836 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3a47f0a3057e46acaf74a5e33bffb30a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465879 7840 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70f60abb54374bb5a1a1475de5feec56 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465878 7852 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467108 7845 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467270 7834 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_700613fee8534a63b231420a5e166fb5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467375 7838 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4bedb57357b24f78a75d1f362473ceff on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467820 7837 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ca14635728c14a7fbd4bce9c90ec208d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463057 7841 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463109 7835 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cf493c1cc4f44b069dba62ecf1ad3676 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463188 7836 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3a47f0a3057e46acaf74a5e33bffb30a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465879 7840 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70f60abb54374bb5a1a1475de5feec56 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465878 7852 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467108 7845 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467270 7834 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_700613fee8534a63b231420a5e166fb5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467375 7838 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4bedb57357b24f78a75d1f362473ceff on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467820 7837 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ca14635728c14a7fbd4bce9c90ec208d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467829 7854 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** SIGABRT (@0x3e800001ad1) received by PID 6865 (TID 0x7f7fef5be700) from PID 6865; stack trace: ***
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463057 7841 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463109 7835 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cf493c1cc4f44b069dba62ecf1ad3676 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463188 7836 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3a47f0a3057e46acaf74a5e33bffb30a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465879 7840 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70f60abb54374bb5a1a1475de5feec56 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465878 7852 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467108 7845 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467270 7834 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_700613fee8534a63b231420a5e166fb5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467375 7838 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4bedb57357b24f78a75d1f362473ceff on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467820 7837 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ca14635728c14a7fbd4bce9c90ec208d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467829 7854 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.468101 7849 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463057 7841 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463109 7835 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cf493c1cc4f44b069dba62ecf1ad3676 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463188 7836 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3a47f0a3057e46acaf74a5e33bffb30a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465879 7840 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70f60abb54374bb5a1a1475de5feec56 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465878 7852 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467108 7845 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467270 7834 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_700613fee8534a63b231420a5e166fb5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467375 7838 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4bedb57357b24f78a75d1f362473ceff on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467820 7837 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ca14635728c14a7fbd4bce9c90ec208d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467829 7854 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.468101 7849 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.469574 7842 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463057 7841 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463109 7835 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cf493c1cc4f44b069dba62ecf1ad3676 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463188 7836 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3a47f0a3057e46acaf74a5e33bffb30a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465879 7840 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70f60abb54374bb5a1a1475de5feec56 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465878 7852 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467108 7845 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467270 7834 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_700613fee8534a63b231420a5e166fb5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467375 7838 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4bedb57357b24f78a75d1f362473ceff on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467820 7837 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ca14635728c14a7fbd4bce9c90ec208d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467829 7854 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.468101 7849 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.469574 7842 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.469669 7855 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.469574 7846 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251029 16:39:28.460965 7850 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460927 7833 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e4dfb9e8508e41669fcd08f8c79b24f8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.460995 7832 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b1bb3149c6d3491f88c5c1c2ae5f8b60 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461308 7853 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461932 7843 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.461933 7844 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462355 7856 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462564 7848 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.462601 7839 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_75040a14be804518a951b603c808d402 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463057 7841 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cafa5404f3ad4912b1a00e3f5ed5dfd4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463109 7835 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cf493c1cc4f44b069dba62ecf1ad3676 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.463188 7836 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3a47f0a3057e46acaf74a5e33bffb30a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465879 7840 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_70f60abb54374bb5a1a1475de5feec56 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.465878 7852 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467108 7845 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467270 7834 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_700613fee8534a63b231420a5e166fb5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467375 7838 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4bedb57357b24f78a75d1f362473ceff on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467820 7837 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ca14635728c14a7fbd4bce9c90ec208d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.467829 7854 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.468101 7849 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.469574 7842 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.469669 7855 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)F20251029 16:39:28.469574 7846 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.180.126:36819,127.6.180.125:44441,127.6.180.124:39271, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.6.180.124:39271: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f800b1dd980 (unknown) at ??:0
@ 0x7f8003fb3fb7 gsignal at ??:0
@ 0x7f8003fb5921 abort at ??:0
@ 0x7f8006cebdcd google::LogMessage::Fail() at ??:0
@ 0x7f8006cefb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f8006ceb7cc google::LogMessage::Flush() at ??:0
@ 0x7f8006cecf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40cb7c kudu::MasterStressTest::DeleteTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:351
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7f800463b6df (unknown) at ??:0
@ 0x7f800b1d26db start_thread at ??:0
@ 0x7f800409671f clone at ??:0