Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250901 19:53:57.582651 3035 test_util.cc:276] Using random seed: -1150598390
2025-09-01T19:53:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T19:53:57Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskaPZSUq/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskaPZSUq/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/security.properties
2025-09-01 19:54:05: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskaPZSUq/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskaPZSUq/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250901 19:54:13.296358 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.254:33523
--webserver_interface=127.2.246.254
--webserver_port=0
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250901 19:54:13.636442 3277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:13.636859 3277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:13.636999 3277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 19:54:13.645918 3277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 19:54:13.646034 3277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:13.646111 3277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 19:54:13.646207 3277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 19:54:13.657122 3277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.254:33523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:13.658919 3277 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:13.660997 3277 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 19:54:13.670706 3283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:13.670818 3282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:13.671113 3285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:13.671566 3277 server_base.cc:1047] running on GCE node
I20250901 19:54:13.672510 3277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:13.674980 3277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:13.676265 3277 hybrid_clock.cc:648] HybridClock initialized: now 1756756453676186 us; error 87 us; skew 500 ppm
I20250901 19:54:13.676721 3277 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:13.679425 3277 webserver.cc:480] Webserver started at http://127.2.246.254:40931/ using document root <none> and password file <none>
I20250901 19:54:13.680392 3277 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:13.680528 3277 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:13.680925 3277 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 19:54:13.683691 3277 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data/instance:
uuid: "4c9163569ae1418090a245256e263186"
format_stamp: "Formatted at 2025-09-01 19:54:13 on dist-test-slave-jnn4"
I20250901 19:54:13.684684 3277 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal/instance:
uuid: "4c9163569ae1418090a245256e263186"
format_stamp: "Formatted at 2025-09-01 19:54:13 on dist-test-slave-jnn4"
I20250901 19:54:13.690876 3277 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250901 19:54:13.695013 3291 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:13.696812 3277 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20250901 19:54:13.697070 3277 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
uuid: "4c9163569ae1418090a245256e263186"
format_stamp: "Formatted at 2025-09-01 19:54:13 on dist-test-slave-jnn4"
I20250901 19:54:13.697317 3277 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:13.730438 3277 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:13.731511 3277 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:13.731860 3277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:13.756390 3277 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.254:33523
I20250901 19:54:13.756410 3342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.254:33523 every 8 connection(s)
I20250901 19:54:13.758160 3277 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data/info.pb
I20250901 19:54:13.762570 3343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:13.767177 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 3277
I20250901 19:54:13.767529 3035 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal/instance
I20250901 19:54:13.771925 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.253:34127
--webserver_interface=127.2.246.253
--webserver_port=0
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250901 19:54:13.771230 3343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } has no permanent_uuid. Determining permanent_uuid...
I20250901 19:54:13.791808 3343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } has no permanent_uuid. Determining permanent_uuid...
W20250901 19:54:13.793198 3294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:54:13.794981 3343 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.2.246.253:34127: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:13.844372 3343 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } attempt: 1
W20250901 19:54:13.846647 3343 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.2.246.253:34127: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:13.914969 3343 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } attempt: 2
W20250901 19:54:13.917243 3343 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.2.246.253:34127: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:14.008581 3343 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } attempt: 3
W20250901 19:54:14.011032 3343 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.2.246.253:34127: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:54:14.145346 3346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:14.145795 3346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:14.145933 3346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 19:54:14.154425 3346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20250901 19:54:14.154356 3343 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } attempt: 4
W20250901 19:54:14.154547 3346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:14.154634 3346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 19:54:14.154707 3346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250901 19:54:14.156599 3343 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.2.246.253:34127: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:14.165777 3346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.253:34127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.246.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:14.167552 3346 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:14.169570 3346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 19:54:14.179502 3353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:14.179538 3352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:14.180197 3355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:14.180585 3346 server_base.cc:1047] running on GCE node
I20250901 19:54:14.181378 3346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:14.182993 3346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:14.184219 3346 hybrid_clock.cc:648] HybridClock initialized: now 1756756454184170 us; error 55 us; skew 500 ppm
I20250901 19:54:14.184618 3346 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:14.187320 3346 webserver.cc:480] Webserver started at http://127.2.246.253:36885/ using document root <none> and password file <none>
I20250901 19:54:14.188230 3346 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:14.188361 3346 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:14.188766 3346 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 19:54:14.191516 3346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/instance:
uuid: "1d944ed54f8c43c9b8c4e7d589dad395"
format_stamp: "Formatted at 2025-09-01 19:54:14 on dist-test-slave-jnn4"
I20250901 19:54:14.192505 3346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal/instance:
uuid: "1d944ed54f8c43c9b8c4e7d589dad395"
format_stamp: "Formatted at 2025-09-01 19:54:14 on dist-test-slave-jnn4"
I20250901 19:54:14.199178 3346 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250901 19:54:14.203424 3361 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:14.205087 3346 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250901 19:54:14.205293 3346 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
uuid: "1d944ed54f8c43c9b8c4e7d589dad395"
format_stamp: "Formatted at 2025-09-01 19:54:14 on dist-test-slave-jnn4"
I20250901 19:54:14.205523 3346 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:14.239214 3346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:14.240272 3346 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:14.240561 3346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:14.263564 3346 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.253:34127
I20250901 19:54:14.263604 3412 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.253:34127 every 8 connection(s)
I20250901 19:54:14.265259 3346 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/info.pb
I20250901 19:54:14.269569 3413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:14.275609 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 3346
I20250901 19:54:14.275875 3035 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal/instance
I20250901 19:54:14.275938 3413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } has no permanent_uuid. Determining permanent_uuid...
I20250901 19:54:14.277118 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.252:37003
--webserver_interface=127.2.246.252
--webserver_port=0
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250901 19:54:14.292395 3413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } has no permanent_uuid. Determining permanent_uuid...
I20250901 19:54:14.303416 3413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } has no permanent_uuid. Determining permanent_uuid...
W20250901 19:54:14.304791 3363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
W20250901 19:54:14.306643 3413 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.2.246.252:37003: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:14.356015 3413 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } attempt: 1
W20250901 19:54:14.358197 3413 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.2.246.252:37003: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:14.426549 3413 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } attempt: 2
W20250901 19:54:14.428670 3413 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.2.246.252:37003: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:14.455955 3343 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } attempt: 5
I20250901 19:54:14.462918 3343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } has no permanent_uuid. Determining permanent_uuid...
W20250901 19:54:14.464967 3343 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.2.246.252:37003: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:14.516283 3343 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } attempt: 1
W20250901 19:54:14.518740 3343 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.2.246.252:37003: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:14.519896 3413 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } attempt: 3
W20250901 19:54:14.521646 3413 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.2.246.252:37003: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:14.587056 3343 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } attempt: 2
W20250901 19:54:14.589339 3343 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.2.246.252:37003: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
W20250901 19:54:14.652758 3415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:14.653119 3415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:14.653213 3415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 19:54:14.662124 3415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 19:54:14.662245 3415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:14.662303 3415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 19:54:14.662353 3415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 19:54:14.664937 3413 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } attempt: 4
W20250901 19:54:14.666883 3413 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.2.246.252:37003: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:14.673524 3415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.252:37003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.246.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:14.675208 3415 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:14.677186 3415 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 19:54:14.685786 3423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:14.685817 3422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:14.686445 3425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:14.687211 3415 server_base.cc:1047] running on GCE node
I20250901 19:54:14.688154 3415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:14.689724 3415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:14.690973 3415 hybrid_clock.cc:648] HybridClock initialized: now 1756756454690909 us; error 62 us; skew 500 ppm
I20250901 19:54:14.691365 3415 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:14.694482 3415 webserver.cc:480] Webserver started at http://127.2.246.252:36357/ using document root <none> and password file <none>
I20250901 19:54:14.695330 3415 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:14.695449 3415 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:14.695623 3343 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } attempt: 3
I20250901 19:54:14.695905 3415 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
W20250901 19:54:14.697616 3343 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.2.246.252:37003: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:14.698757 3415 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/instance:
uuid: "08076b584d2a4aec8fd9868f85c9c7cd"
format_stamp: "Formatted at 2025-09-01 19:54:14 on dist-test-slave-jnn4"
I20250901 19:54:14.699681 3415 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal/instance:
uuid: "08076b584d2a4aec8fd9868f85c9c7cd"
format_stamp: "Formatted at 2025-09-01 19:54:14 on dist-test-slave-jnn4"
I20250901 19:54:14.705824 3415 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250901 19:54:14.710290 3431 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:14.712561 3415 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250901 19:54:14.712814 3415 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
uuid: "08076b584d2a4aec8fd9868f85c9c7cd"
format_stamp: "Formatted at 2025-09-01 19:54:14 on dist-test-slave-jnn4"
I20250901 19:54:14.713110 3415 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:14.729027 3415 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:14.729985 3415 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:14.730273 3415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:14.753695 3415 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.252:37003
I20250901 19:54:14.753708 3482 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.252:37003 every 8 connection(s)
I20250901 19:54:14.755401 3415 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
I20250901 19:54:14.759537 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 3415
I20250901 19:54:14.759835 3035 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal/instance
I20250901 19:54:14.760823 3483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:14.769125 3483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } has no permanent_uuid. Determining permanent_uuid...
I20250901 19:54:14.783871 3483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } has no permanent_uuid. Determining permanent_uuid...
I20250901 19:54:14.793851 3483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } has no permanent_uuid. Determining permanent_uuid...
I20250901 19:54:14.806354 3483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Bootstrap starting.
I20250901 19:54:14.809934 3483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:14.811067 3483 log.cc:826] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:14.813819 3483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: No bootstrap required, opened a new log
I20250901 19:54:14.819813 3483 raft_consensus.cc:357] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:14.820183 3483 raft_consensus.cc:383] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:14.820293 3483 raft_consensus.cc:738] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08076b584d2a4aec8fd9868f85c9c7cd, State: Initialized, Role: FOLLOWER
I20250901 19:54:14.821053 3483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:14.822749 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:14.823017 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:14.824003 3483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: configured and running, proceeding with master startup.
W20250901 19:54:14.836669 3505 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250901 19:54:14.836838 3505 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250901 19:54:14.833722 3415 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: OK, instance_id { permanent_uuid: "4c9163569ae1418090a245256e263186" instance_seqno: 1756756453749882 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250901 19:54:14.840387 3415 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:54:14.874871 3343 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } attempt: 4
I20250901 19:54:14.892343 3343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Bootstrap starting.
I20250901 19:54:14.897109 3343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:14.898723 3343 log.cc:826] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:14.909031 3343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: No bootstrap required, opened a new log
I20250901 19:54:14.917125 3343 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:14.917572 3343 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:14.917716 3343 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9163569ae1418090a245256e263186, State: Initialized, Role: FOLLOWER
I20250901 19:54:14.918675 3343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:14.923784 3509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:14.924703 3509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:14.928833 3343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 19:54:14.937984 3491 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:14.938288 3491 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:14.940191 3491 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
I20250901 19:54:14.941535 3318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9163569ae1418090a245256e263186" is_pre_election: true
I20250901 19:54:14.942253 3318 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08076b584d2a4aec8fd9868f85c9c7cd in term 0.
I20250901 19:54:14.943039 3434 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186; no voters:
I20250901 19:54:14.943527 3491 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:14.943728 3491 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:14.943825 3491 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:14.947093 3491 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:14.948534 3318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9163569ae1418090a245256e263186"
I20250901 19:54:14.948843 3318 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:14.949062 3491 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 1 election: Requested vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
I20250901 19:54:14.953711 3318 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08076b584d2a4aec8fd9868f85c9c7cd in term 1.
I20250901 19:54:14.954516 3434 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186; no voters:
I20250901 19:54:14.955070 3491 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:14.958995 3491 raft_consensus.cc:695] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 1 LEADER]: Becoming Leader. State: Replica: 08076b584d2a4aec8fd9868f85c9c7cd, State: Running, Role: LEADER
I20250901 19:54:14.960073 3491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:14.966085 3413 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } attempt: 5
W20250901 19:54:14.982482 3437 tablet.cc:2378] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 19:54:14.984225 3413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Bootstrap starting.
W20250901 19:54:14.985102 3297 tablet.cc:2378] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 19:54:14.993438 3413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:14.999189 3521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08076b584d2a4aec8fd9868f85c9c7cd. Latest consensus state: current_term: 1 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:14.999585 3521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:15.006913 3277 master_runner.cc:186] Error getting master registration for 127.2.246.253:34127: OK, instance_id { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" instance_seqno: 1756756454257458 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250901 19:54:15.014477 3277 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:54:15.015488 3413 log.cc:826] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:15.017127 3527 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 19:54:15.028542 3527 catalog_manager.cc:1486] Initializing Kudu cluster ID...
W20250901 19:54:15.028475 3526 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250901 19:54:15.029604 3526 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250901 19:54:15.030380 3413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: No bootstrap required, opened a new log
I20250901 19:54:15.039165 3413 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:15.039840 3413 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:15.040042 3413 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d944ed54f8c43c9b8c4e7d589dad395, State: Initialized, Role: FOLLOWER
I20250901 19:54:15.041222 3413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:15.048489 3531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:15.051452 3531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:15.051347 3387 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d944ed54f8c43c9b8c4e7d589dad395"
I20250901 19:54:15.051286 3413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 19:54:15.051894 3386 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:15.052424 3388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" is_pre_election: true
I20250901 19:54:15.054934 3318 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 1 FOLLOWER]: Refusing update from remote peer 08076b584d2a4aec8fd9868f85c9c7cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 19:54:15.056020 3521 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:15.056224 3386 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 1 FOLLOWER]: Refusing update from remote peer 08076b584d2a4aec8fd9868f85c9c7cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 19:54:15.057286 3491 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:15.076597 3509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08076b584d2a4aec8fd9868f85c9c7cd. Latest consensus state: current_term: 1 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:15.076953 3509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:15.079113 3531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08076b584d2a4aec8fd9868f85c9c7cd. Latest consensus state: current_term: 1 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:15.079396 3531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:15.086751 3530 mvcc.cc:204] Tried to move back new op lower bound from 7195674439880065024 to 7195674439528980480. Current Snapshot: MvccSnapshot[applied={T|T < 7195674439880065024}]
I20250901 19:54:15.088788 3509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:15.089082 3509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:15.092583 3531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:15.092821 3531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:15.104926 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:15.105307 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:15.105633 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:15.105880 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:15.109124 3527 catalog_manager.cc:1349] Generated new cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:15.109290 3527 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 19:54:15.151213 3527 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 19:54:15.157704 3527 catalog_manager.cc:1506] Loading token signing keys...
I20250901 19:54:15.171101 3554 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250901 19:54:15.198550 3527 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Generated new TSK 0
I20250901 19:54:15.199957 3527 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 19:54:15.201431 3527 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 19:54:15.202766 3527 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250901 19:54:15.218612 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.193:0
--local_ip_for_outbound_sockets=127.2.246.193
--webserver_interface=127.2.246.193
--webserver_port=0
--tserver_master_addrs=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250901 19:54:15.675411 3555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:15.675808 3555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:15.675959 3555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 19:54:15.685473 3555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:15.685698 3555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.193
I20250901 19:54:15.696223 3555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.246.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.193
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:15.697955 3555 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:15.699656 3559 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:15.699879 3559 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:15.700188 3555 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250901 19:54:15.701438 3559 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
I20250901 19:54:15.702023 3386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c9163569ae1418090a245256e263186" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" is_pre_election: true
I20250901 19:54:15.702165 3458 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c9163569ae1418090a245256e263186" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" is_pre_election: true
I20250901 19:54:15.703117 3293 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c9163569ae1418090a245256e263186; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395
I20250901 19:54:15.703751 3559 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250901 19:54:15.711813 3561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:15.711709 3562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:15.712957 3564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:15.714118 3555 server_base.cc:1047] running on GCE node
I20250901 19:54:15.714824 3555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:15.715979 3555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:15.717242 3555 hybrid_clock.cc:648] HybridClock initialized: now 1756756455717187 us; error 54 us; skew 500 ppm
I20250901 19:54:15.717646 3555 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:15.720580 3555 webserver.cc:480] Webserver started at http://127.2.246.193:36223/ using document root <none> and password file <none>
I20250901 19:54:15.721446 3555 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:15.721561 3555 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:15.721940 3555 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 19:54:15.724670 3555 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/data/instance:
uuid: "c5b48d93671f4eb8980759b876e9ca21"
format_stamp: "Formatted at 2025-09-01 19:54:15 on dist-test-slave-jnn4"
I20250901 19:54:15.725571 3555 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/wal/instance:
uuid: "c5b48d93671f4eb8980759b876e9ca21"
format_stamp: "Formatted at 2025-09-01 19:54:15 on dist-test-slave-jnn4"
I20250901 19:54:15.732009 3555 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250901 19:54:15.736315 3570 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:15.738085 3555 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250901 19:54:15.738278 3555 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/wal
uuid: "c5b48d93671f4eb8980759b876e9ca21"
format_stamp: "Formatted at 2025-09-01 19:54:15 on dist-test-slave-jnn4"
I20250901 19:54:15.738499 3555 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:15.762037 3555 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:15.763020 3555 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:15.763370 3555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:15.765036 3555 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 19:54:15.767128 3555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 19:54:15.767236 3555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:15.767398 3555 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 19:54:15.767489 3555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:15.811949 3555 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.193:37767
I20250901 19:54:15.812031 3682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.193:37767 every 8 connection(s)
I20250901 19:54:15.813535 3555 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/data/info.pb
I20250901 19:54:15.818902 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 3555
I20250901 19:54:15.819203 3035 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/wal/instance
I20250901 19:54:15.844338 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.194:0
--local_ip_for_outbound_sockets=127.2.246.194
--webserver_interface=127.2.246.194
--webserver_port=0
--tserver_master_addrs=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250901 19:54:15.854239 3685 heartbeater.cc:344] Connected to a master server at 127.2.246.253:34127
I20250901 19:54:15.855175 3685 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:15.857062 3685 heartbeater.cc:507] Master 127.2.246.253:34127 requested a full tablet report, sending...
I20250901 19:54:15.858202 3683 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
I20250901 19:54:15.858399 3683 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:15.859066 3683 heartbeater.cc:507] Master 127.2.246.252:37003 requested a full tablet report, sending...
I20250901 19:54:15.861274 3378 ts_manager.cc:194] Registered new tserver with Master: c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:15.861275 3448 ts_manager.cc:194] Registered new tserver with Master: c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:15.863524 3448 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.193:37597
I20250901 19:54:15.867985 3684 heartbeater.cc:344] Connected to a master server at 127.2.246.254:33523
I20250901 19:54:15.868193 3684 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:15.868817 3684 heartbeater.cc:507] Master 127.2.246.254:33523 requested a full tablet report, sending...
I20250901 19:54:15.871030 3308 ts_manager.cc:194] Registered new tserver with Master: c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:16.031945 3526 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:16.032059 3526 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: loading cluster ID for follower catalog manager: success
I20250901 19:54:16.034952 3526 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: acquiring CA information for follower catalog manager: success
I20250901 19:54:16.036911 3526 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 19:54:16.178265 3554 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:16.178498 3554 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: loading cluster ID for follower catalog manager: success
I20250901 19:54:16.183501 3554 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: acquiring CA information for follower catalog manager: success
I20250901 19:54:16.187561 3554 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 19:54:16.192224 3695 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:16.192652 3695 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:16.194630 3695 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
I20250901 19:54:16.195399 3318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c9163569ae1418090a245256e263186" is_pre_election: true
I20250901 19:54:16.195602 3458 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" is_pre_election: true
I20250901 19:54:16.196467 3364 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186
I20250901 19:54:16.197317 3695 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250901 19:54:16.289220 3693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:16.289892 3693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:16.290200 3693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250901 19:54:16.304023 3699 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:16.304435 3699 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:16.305820 3386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c9163569ae1418090a245256e263186" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" is_pre_election: true
I20250901 19:54:16.307091 3699 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
I20250901 19:54:16.307430 3458 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c9163569ae1418090a245256e263186" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" is_pre_election: true
I20250901 19:54:16.308074 3293 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c9163569ae1418090a245256e263186; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395
W20250901 19:54:16.308174 3693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:16.308399 3693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.194
I20250901 19:54:16.308624 3699 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250901 19:54:16.319891 3693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.246.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.194
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:16.321805 3693 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:16.323856 3693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 19:54:16.336208 3701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:16.344624 3702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:16.346846 3693 server_base.cc:1047] running on GCE node
W20250901 19:54:16.347177 3704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:16.348126 3693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:16.349666 3693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:16.351313 3693 hybrid_clock.cc:648] HybridClock initialized: now 1756756456351236 us; error 54 us; skew 500 ppm
I20250901 19:54:16.351843 3693 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:16.355232 3693 webserver.cc:480] Webserver started at http://127.2.246.194:40353/ using document root <none> and password file <none>
I20250901 19:54:16.356630 3693 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:16.356794 3693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:16.357264 3693 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 19:54:16.361711 3693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/data/instance:
uuid: "d397574faadb441192c99d530f14a734"
format_stamp: "Formatted at 2025-09-01 19:54:16 on dist-test-slave-jnn4"
I20250901 19:54:16.363075 3693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/wal/instance:
uuid: "d397574faadb441192c99d530f14a734"
format_stamp: "Formatted at 2025-09-01 19:54:16 on dist-test-slave-jnn4"
I20250901 19:54:16.367516 3683 heartbeater.cc:499] Master 127.2.246.252:37003 was elected leader, sending a full tablet report...
I20250901 19:54:16.372864 3693 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20250901 19:54:16.378893 3710 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:16.381176 3693 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20250901 19:54:16.381534 3693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/wal
uuid: "d397574faadb441192c99d530f14a734"
format_stamp: "Formatted at 2025-09-01 19:54:16 on dist-test-slave-jnn4"
I20250901 19:54:16.381889 3693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:16.412266 3693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:16.413698 3693 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:16.414142 3693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:16.416452 3693 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 19:54:16.419118 3693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 19:54:16.419258 3693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:16.419416 3693 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 19:54:16.419494 3693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:16.471699 3693 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.194:33557
I20250901 19:54:16.471889 3822 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.194:33557 every 8 connection(s)
I20250901 19:54:16.473878 3693 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/data/info.pb
I20250901 19:54:16.480185 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 3693
I20250901 19:54:16.480413 3035 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/wal/instance
I20250901 19:54:16.499389 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.246.195:0
--local_ip_for_outbound_sockets=127.2.246.195
--webserver_interface=127.2.246.195
--webserver_port=0
--tserver_master_addrs=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250901 19:54:16.523524 3824 heartbeater.cc:344] Connected to a master server at 127.2.246.254:33523
I20250901 19:54:16.523963 3824 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:16.525197 3824 heartbeater.cc:507] Master 127.2.246.254:33523 requested a full tablet report, sending...
I20250901 19:54:16.526607 3823 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
I20250901 19:54:16.526767 3823 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:16.527316 3823 heartbeater.cc:507] Master 127.2.246.252:37003 requested a full tablet report, sending...
I20250901 19:54:16.527496 3308 ts_manager.cc:194] Registered new tserver with Master: d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:16.528373 3448 ts_manager.cc:194] Registered new tserver with Master: d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:16.529448 3448 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.194:37487
I20250901 19:54:16.535149 3829 heartbeater.cc:344] Connected to a master server at 127.2.246.253:34127
I20250901 19:54:16.535320 3829 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:16.535856 3829 heartbeater.cc:507] Master 127.2.246.253:34127 requested a full tablet report, sending...
I20250901 19:54:16.537317 3378 ts_manager.cc:194] Registered new tserver with Master: d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:16.752480 3834 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:16.752741 3834 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:16.753496 3834 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
I20250901 19:54:16.754065 3318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c9163569ae1418090a245256e263186" is_pre_election: true
I20250901 19:54:16.754281 3458 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" is_pre_election: true
I20250901 19:54:16.754854 3363 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186
I20250901 19:54:16.755164 3834 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250901 19:54:16.893440 3830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:16.893755 3830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:16.893854 3830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 19:54:16.902839 3830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:16.903110 3830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.246.195
I20250901 19:54:16.913981 3830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.246.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.246.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.246.195
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:16.915814 3830 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:16.917816 3830 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 19:54:16.931820 3839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:16.934093 3840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:16.936149 3842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:16.937346 3830 server_base.cc:1047] running on GCE node
I20250901 19:54:16.938248 3830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:16.939805 3830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:16.941128 3830 hybrid_clock.cc:648] HybridClock initialized: now 1756756456941055 us; error 60 us; skew 500 ppm
I20250901 19:54:16.941622 3830 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:16.944761 3830 webserver.cc:480] Webserver started at http://127.2.246.195:38371/ using document root <none> and password file <none>
I20250901 19:54:16.945998 3830 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:16.946166 3830 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:16.946750 3830 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 19:54:16.950681 3830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/data/instance:
uuid: "e0064a1b18e5470b85bde2277757aac8"
format_stamp: "Formatted at 2025-09-01 19:54:16 on dist-test-slave-jnn4"
I20250901 19:54:16.951952 3830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/wal/instance:
uuid: "e0064a1b18e5470b85bde2277757aac8"
format_stamp: "Formatted at 2025-09-01 19:54:16 on dist-test-slave-jnn4"
I20250901 19:54:16.960021 3830 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20250901 19:54:16.964174 3848 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:16.965783 3830 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250901 19:54:16.965978 3830 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/wal
uuid: "e0064a1b18e5470b85bde2277757aac8"
format_stamp: "Formatted at 2025-09-01 19:54:16 on dist-test-slave-jnn4"
I20250901 19:54:16.966204 3830 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:16.986160 3830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:16.987134 3830 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:16.987435 3830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:16.988548 3830 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 19:54:16.990427 3830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 19:54:16.990558 3830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:16.990669 3830 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 19:54:16.990736 3830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:17.032004 3830 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.195:33271
I20250901 19:54:17.032044 3960 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.195:33271 every 8 connection(s)
I20250901 19:54:17.032418 3823 heartbeater.cc:499] Master 127.2.246.252:37003 was elected leader, sending a full tablet report...
I20250901 19:54:17.033854 3830 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/data/info.pb
I20250901 19:54:17.039572 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 3830
I20250901 19:54:17.039872 3035 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/wal/instance
I20250901 19:54:17.061152 3962 heartbeater.cc:344] Connected to a master server at 127.2.246.254:33523
I20250901 19:54:17.061306 3961 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
I20250901 19:54:17.061633 3962 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:17.061630 3961 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:17.062865 3961 heartbeater.cc:507] Master 127.2.246.252:37003 requested a full tablet report, sending...
I20250901 19:54:17.064009 3963 heartbeater.cc:344] Connected to a master server at 127.2.246.253:34127
I20250901 19:54:17.064148 3962 heartbeater.cc:507] Master 127.2.246.254:33523 requested a full tablet report, sending...
I20250901 19:54:17.064175 3963 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:17.064736 3448 ts_manager.cc:194] Registered new tserver with Master: e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:17.064813 3963 heartbeater.cc:507] Master 127.2.246.253:34127 requested a full tablet report, sending...
I20250901 19:54:17.065356 3308 ts_manager.cc:194] Registered new tserver with Master: e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:17.065701 3448 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.246.195:57277
I20250901 19:54:17.065939 3378 ts_manager.cc:194] Registered new tserver with Master: e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:17.074175 3035 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250901 19:54:17.109445 3448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35034:
name: "default.Table_079afb395d5c4c17a18fd7e77548109d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.109405 3447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35034:
name: "default.Table_c32254b687594ed89daf3edaded35200"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.111456 3445 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35034:
name: "default.Table_11ed1535329246fc866753e35e0cb6a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.111703 3446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35034:
name: "default.Table_ea3686ff3ba94b9a984657e263ca0832"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.112840 3440 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35034:
name: "default.Table_0a1ca85b493d4aa285d85b17df52c4aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.113193 3443 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35034:
name: "default.Table_3cbad8a0189443d9b60b0abc4460862a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.113859 3442 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35034:
name: "default.Table_51c52d3eec714bbd9a77bed453c02bb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.114391 3441 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35034:
name: "default.Table_d29d54fc138b40009682d5b2585d25bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.115275 3444 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35034:
name: "default.Table_69ed2a785efc4a4f8077c8aa2ef89c35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.115803 3439 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35034:
name: "default.Table_5cf2893927ed42d597f83e0a04da2245"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 19:54:17.118665 3447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c32254b687594ed89daf3edaded35200 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 19:54:17.122478 3448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_079afb395d5c4c17a18fd7e77548109d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 19:54:17.145026 3035 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu pid 3415
I20250901 19:54:17.146548 3478 generic_service.cc:196] Checking for leaks (request via RPC)
I20250901 19:54:17.292255 4011 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:17.293668 4011 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:17.295066 3035 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu with pid 3415
I20250901 19:54:17.296276 4011 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
I20250901 19:54:17.296887 3318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c9163569ae1418090a245256e263186" is_pre_election: true
W20250901 19:54:17.333384 3972 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.246.252:37003)
W20250901 19:54:17.336598 3363 connection.cc:537] client connection to 127.2.246.252:37003 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:17.338611 3976 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 19:54:17.335143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.252:37003 (local address 127.0.0.1:35036)
0901 19:54:17.335506 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0901 19:54:17.335528 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0901 19:54:17.335759 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 19:54:17.336929 (+ 1170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":86}
I20250901 19:54:17.350070 4014 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
W20250901 19:54:17.351586 3363 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:17.351836 3363 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186
I20250901 19:54:17.352188 4014 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:17.353240 4014 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
I20250901 19:54:17.353495 4011 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250901 19:54:17.354805 3386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c9163569ae1418090a245256e263186" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" is_pre_election: true
I20250901 19:54:17.355072 3386 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c9163569ae1418090a245256e263186 in term 1.
I20250901 19:54:17.355733 3294 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186; no voters:
I20250901 19:54:17.356246 4014 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250901 19:54:17.356468 4014 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:17.356653 4014 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 1 FOLLOWER]: Advancing to term 2
I20250901 19:54:17.361048 4014 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:17.362516 3386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c9163569ae1418090a245256e263186" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1d944ed54f8c43c9b8c4e7d589dad395"
I20250901 19:54:17.362756 3386 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 1 FOLLOWER]: Advancing to term 2
I20250901 19:54:17.366050 3386 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c9163569ae1418090a245256e263186 in term 2.
I20250901 19:54:17.369632 3294 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186; no voters:
I20250901 19:54:17.370910 4014 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 2 election: Requested vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
I20250901 19:54:17.371562 4014 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 2 FOLLOWER]: Leader election won for term 2
I20250901 19:54:17.371824 4014 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 2 LEADER]: Becoming Leader. State: Replica: 4c9163569ae1418090a245256e263186, State: Running, Role: LEADER
I20250901 19:54:17.372620 4014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
W20250901 19:54:17.397596 3683 heartbeater.cc:646] Failed to heartbeat to 127.2.246.252:37003 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:17.401216 4017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c9163569ae1418090a245256e263186. Latest consensus state: current_term: 2 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:17.401600 4017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: LEADER
W20250901 19:54:17.403604 3293 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
W20250901 19:54:17.404011 3293 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:17.404023 4021 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 19:54:17.406785 4021 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 19:54:17.407932 4021 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:17.408038 4021 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 19:54:17.409626 4021 catalog_manager.cc:1506] Loading token signing keys...
I20250901 19:54:17.410733 4021 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Loaded TSK: 0
I20250901 19:54:17.412082 4021 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 19:54:17.413040 4021 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 19:54:17.413846 4021 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250901 19:54:17.427064 3307 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55634:
name: "default.Table_0a1ca85b493d4aa285d85b17df52c4aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.428485 3308 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55634:
name: "default.Table_5cf2893927ed42d597f83e0a04da2245"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.429222 3306 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55634:
name: "default.Table_d29d54fc138b40009682d5b2585d25bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.430076 3304 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55634:
name: "default.Table_c32254b687594ed89daf3edaded35200"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.430490 3305 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55634:
name: "default.Table_51c52d3eec714bbd9a77bed453c02bb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.431195 3303 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55634:
name: "default.Table_079afb395d5c4c17a18fd7e77548109d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.431288 3302 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55634:
name: "default.Table_ea3686ff3ba94b9a984657e263ca0832"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.432221 3301 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55634:
name: "default.Table_11ed1535329246fc866753e35e0cb6a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.432404 3300 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55634:
name: "default.Table_69ed2a785efc4a4f8077c8aa2ef89c35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:17.432468 3299 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55634:
name: "default.Table_3cbad8a0189443d9b60b0abc4460862a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 19:54:17.441658 3307 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0a1ca85b493d4aa285d85b17df52c4aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 19:54:17.446146 3308 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5cf2893927ed42d597f83e0a04da2245 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 19:54:17.447350 3306 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d29d54fc138b40009682d5b2585d25bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 19:54:17.448459 3302 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ea3686ff3ba94b9a984657e263ca0832 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 19:54:17.449503 3300 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 19:54:17.450498 3299 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3cbad8a0189443d9b60b0abc4460862a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 19:54:17.451658 3303 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_079afb395d5c4c17a18fd7e77548109d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 19:54:17.452682 3301 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_11ed1535329246fc866753e35e0cb6a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 19:54:17.453740 3305 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_51c52d3eec714bbd9a77bed453c02bb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 19:54:17.446152 3304 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c32254b687594ed89daf3edaded35200 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 19:54:17.546067 3823 heartbeater.cc:646] Failed to heartbeat to 127.2.246.252:37003 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:17.567945 3961 heartbeater.cc:499] Master 127.2.246.252:37003 was elected leader, sending a full tablet report...
W20250901 19:54:17.569365 3961 heartbeater.cc:646] Failed to heartbeat to 127.2.246.252:37003 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:17.692759 3307 catalog_manager.cc:2149] created HMS catalog entry for table [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]
I20250901 19:54:17.710233 3386 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 2 FOLLOWER]: Refusing update from remote peer 4c9163569ae1418090a245256e263186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250901 19:54:17.711010 4014 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250901 19:54:17.712524 3293 consensus_peers.cc:489] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 -> Peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Couldn't send request to peer 08076b584d2a4aec8fd9868f85c9c7cd. Status: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 19:54:17.728422 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c9163569ae1418090a245256e263186. Latest consensus state: current_term: 2 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:17.728691 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:17.731225 4024 mvcc.cc:204] Tried to move back new op lower bound from 7195674450773413888 to 7195674449407266816. Current Snapshot: MvccSnapshot[applied={T|T < 7195674450773413888}]
I20250901 19:54:17.734503 4025 mvcc.cc:204] Tried to move back new op lower bound from 7195674450773413888 to 7195674449407266816. Current Snapshot: MvccSnapshot[applied={T|T < 7195674450773413888}]
I20250901 19:54:17.737495 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:17.737797 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:17.739460 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:17.739686 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:17.745064 3307 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55634:
table { table_name: "default.Table_0a1ca85b493d4aa285d85b17df52c4aa" } new_table_name: "default.Table_ed1ee4c8517b4614aafb4f78827e4f6e" modify_external_catalogs: true
I20250901 19:54:17.747583 3308 catalog_manager.cc:2149] created HMS catalog entry for table [id=9a7c50e73c39409f82c0c4339e7af8b3]
I20250901 19:54:17.786983 3896 tablet_service.cc:1468] Processing CreateTablet for tablet 1a600536db3a40b28f031061d02a49a6 (DEFAULT_TABLE table=default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.789355 3896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a600536db3a40b28f031061d02a49a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:17.794132 3618 tablet_service.cc:1468] Processing CreateTablet for tablet 1a600536db3a40b28f031061d02a49a6 (DEFAULT_TABLE table=default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.796195 3618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a600536db3a40b28f031061d02a49a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:17.801875 3758 tablet_service.cc:1468] Processing CreateTablet for tablet 1a600536db3a40b28f031061d02a49a6 (DEFAULT_TABLE table=default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.803548 3758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a600536db3a40b28f031061d02a49a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:17.821751 3308 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55634:
table { table_name: "default.Table_5cf2893927ed42d597f83e0a04da2245" } new_table_name: "default.Table_3d911927cc9648b39bc8fae3c7d25c87" modify_external_catalogs: true
I20250901 19:54:17.823099 3306 catalog_manager.cc:2149] created HMS catalog entry for table [id=57841a49107e4375aa82904e212fc896]
I20250901 19:54:17.830883 3757 tablet_service.cc:1468] Processing CreateTablet for tablet f640cc3c97394e16b88f7ef8148c19dd (DEFAULT_TABLE table=default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.831023 3617 tablet_service.cc:1468] Processing CreateTablet for tablet f640cc3c97394e16b88f7ef8148c19dd (DEFAULT_TABLE table=default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.831043 3895 tablet_service.cc:1468] Processing CreateTablet for tablet f640cc3c97394e16b88f7ef8148c19dd (DEFAULT_TABLE table=default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.831544 3617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f640cc3c97394e16b88f7ef8148c19dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:17.831566 3757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f640cc3c97394e16b88f7ef8148c19dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:17.831568 3895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f640cc3c97394e16b88f7ef8148c19dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:17.841850 4037 tablet_bootstrap.cc:492] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:17.844704 3306 master_service.cc:948] ReplaceTablet: received request to replace tablet 1a600536db3a40b28f031061d02a49a6 from {username='slave'} at 127.0.0.1:55634
I20250901 19:54:17.852633 3617 tablet_service.cc:1468] Processing CreateTablet for tablet 1d21a96219ab4f5f96bc6d9976f990ff (DEFAULT_TABLE table=default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.852752 4038 tablet_bootstrap.cc:492] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:17.852705 3895 tablet_service.cc:1468] Processing CreateTablet for tablet 1d21a96219ab4f5f96bc6d9976f990ff (DEFAULT_TABLE table=default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.853107 3617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d21a96219ab4f5f96bc6d9976f990ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:17.853159 3895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d21a96219ab4f5f96bc6d9976f990ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:17.856035 3758 tablet_service.cc:1468] Processing CreateTablet for tablet 1d21a96219ab4f5f96bc6d9976f990ff (DEFAULT_TABLE table=default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.856756 3758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d21a96219ab4f5f96bc6d9976f990ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:17.857556 4038 tablet_bootstrap.cc:654] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:17.859009 4038 log.cc:826] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:17.862197 3306 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Sending DeleteTablet for 3 replicas of tablet 1a600536db3a40b28f031061d02a49a6
I20250901 19:54:17.863096 3306 catalog_manager.cc:6652] ReplaceTablet: tablet 1a600536db3a40b28f031061d02a49a6 deleted and replaced by tablet 0d8339cbcc774ea5bae71abeb3ecd2b7
I20250901 19:54:17.863425 3306 master_service.cc:948] ReplaceTablet: received request to replace tablet 1a600536db3a40b28f031061d02a49a6 from {username='slave'} at 127.0.0.1:55634
I20250901 19:54:17.863960 3306 master_service.cc:948] ReplaceTablet: received request to replace tablet 1a600536db3a40b28f031061d02a49a6 from {username='slave'} at 127.0.0.1:55634
I20250901 19:54:17.864349 3306 master_service.cc:948] ReplaceTablet: received request to replace tablet 1a600536db3a40b28f031061d02a49a6 from {username='slave'} at 127.0.0.1:55634
I20250901 19:54:17.864570 4036 tablet_bootstrap.cc:492] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:17.864828 3306 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55634:
table { table_name: "default.Table_d29d54fc138b40009682d5b2585d25bd" } new_table_name: "default.Table_6a818be47b2e41e48156df825fbe0cc3" modify_external_catalogs: true
I20250901 19:54:17.866518 3616 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a600536db3a40b28f031061d02a49a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d8339cbcc774ea5bae71abeb3ecd2b7) from {username='slave'} at 127.0.0.1:43308
I20250901 19:54:17.866650 3896 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a600536db3a40b28f031061d02a49a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d8339cbcc774ea5bae71abeb3ecd2b7) from {username='slave'} at 127.0.0.1:38840
I20250901 19:54:17.869454 3757 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a600536db3a40b28f031061d02a49a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d8339cbcc774ea5bae71abeb3ecd2b7) from {username='slave'} at 127.0.0.1:52658
I20250901 19:54:17.874980 4037 tablet_bootstrap.cc:654] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:17.877398 4037 log.cc:826] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:17.881922 3895 tablet_service.cc:1468] Processing CreateTablet for tablet 0d8339cbcc774ea5bae71abeb3ecd2b7 (DEFAULT_TABLE table=default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.882649 3895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d8339cbcc774ea5bae71abeb3ecd2b7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 19:54:17.882602 3294 catalog_manager.cc:4908] TS d397574faadb441192c99d530f14a734 (127.2.246.194:33557): delete failed for tablet 1a600536db3a40b28f031061d02a49a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a600536db3a40b28f031061d02a49a6 already in progress: creating tablet
I20250901 19:54:17.882937 3618 tablet_service.cc:1468] Processing CreateTablet for tablet 0d8339cbcc774ea5bae71abeb3ecd2b7 (DEFAULT_TABLE table=default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.883306 3758 tablet_service.cc:1468] Processing CreateTablet for tablet 0d8339cbcc774ea5bae71abeb3ecd2b7 (DEFAULT_TABLE table=default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.883895 3618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d8339cbcc774ea5bae71abeb3ecd2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:17.885046 3758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d8339cbcc774ea5bae71abeb3ecd2b7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 19:54:17.889343 3293 catalog_manager.cc:4908] TS e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271): delete failed for tablet 1a600536db3a40b28f031061d02a49a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a600536db3a40b28f031061d02a49a6 already in progress: creating tablet
I20250901 19:54:17.893118 4037 tablet_bootstrap.cc:492] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:17.893476 4037 ts_tablet_manager.cc:1397] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.004s
I20250901 19:54:17.897024 4036 tablet_bootstrap.cc:654] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:17.898211 4038 tablet_bootstrap.cc:492] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:17.898520 4038 ts_tablet_manager.cc:1397] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.000s
W20250901 19:54:17.901316 3292 catalog_manager.cc:4908] TS c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): delete failed for tablet 1a600536db3a40b28f031061d02a49a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a600536db3a40b28f031061d02a49a6 already in progress: creating tablet
I20250901 19:54:17.903244 4036 log.cc:826] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:17.907145 4038 raft_consensus.cc:357] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:17.907595 4038 raft_consensus.cc:383] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:17.907752 4038 raft_consensus.cc:738] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:17.908181 4037 raft_consensus.cc:357] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:17.908561 4037 raft_consensus.cc:383] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:17.908690 4037 raft_consensus.cc:738] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:17.908581 4038 consensus_queue.cc:260] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:17.916914 4038 ts_tablet_manager.cc:1428] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20250901 19:54:17.917966 3962 heartbeater.cc:499] Master 127.2.246.254:33523 was elected leader, sending a full tablet report...
I20250901 19:54:17.919576 4037 consensus_queue.cc:260] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:17.921316 4036 tablet_bootstrap.cc:492] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:17.921672 4036 ts_tablet_manager.cc:1397] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.017s
I20250901 19:54:17.922610 4038 tablet_bootstrap.cc:492] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:17.927244 4038 tablet_bootstrap.cc:654] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:17.928972 3824 heartbeater.cc:499] Master 127.2.246.254:33523 was elected leader, sending a full tablet report...
I20250901 19:54:17.928885 4036 raft_consensus.cc:357] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:17.929332 4036 raft_consensus.cc:383] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:17.929451 4036 raft_consensus.cc:738] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:17.930029 4037 ts_tablet_manager.cc:1428] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.036s user 0.013s sys 0.000s
I20250901 19:54:17.930176 4036 consensus_queue.cc:260] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:17.930459 4037 tablet_bootstrap.cc:492] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:17.932610 4037 tablet_bootstrap.cc:654] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:17.939671 4038 tablet_bootstrap.cc:492] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:17.939884 4038 ts_tablet_manager.cc:1397] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20250901 19:54:17.939951 4036 ts_tablet_manager.cc:1428] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.018s user 0.007s sys 0.013s
I20250901 19:54:17.940127 4037 tablet_bootstrap.cc:492] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:17.941259 4038 raft_consensus.cc:357] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:17.941639 4038 raft_consensus.cc:383] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:17.941728 4038 raft_consensus.cc:738] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:17.942013 4038 consensus_queue.cc:260] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:17.942741 4036 tablet_bootstrap.cc:492] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:17.942958 3302 catalog_manager.cc:2149] created HMS catalog entry for table [id=e3dcc8f77c32491487c240aeadfb3616]
I20250901 19:54:17.945360 4038 ts_tablet_manager.cc:1428] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250901 19:54:17.945817 4038 tablet_bootstrap.cc:492] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:17.947587 4036 tablet_bootstrap.cc:654] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:17.950064 4038 tablet_bootstrap.cc:654] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:17.940299 4037 ts_tablet_manager.cc:1397] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20250901 19:54:17.952023 4037 raft_consensus.cc:357] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:17.952316 4037 raft_consensus.cc:383] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:17.952409 4037 raft_consensus.cc:738] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:17.952770 4037 consensus_queue.cc:260] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:17.954861 4037 ts_tablet_manager.cc:1428] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250901 19:54:17.956323 3895 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a600536db3a40b28f031061d02a49a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d8339cbcc774ea5bae71abeb3ecd2b7) from {username='slave'} at 127.0.0.1:38840
I20250901 19:54:17.957073 3684 heartbeater.cc:499] Master 127.2.246.254:33523 was elected leader, sending a full tablet report...
I20250901 19:54:17.957566 4040 tablet_replica.cc:331] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8: stopping tablet replica
I20250901 19:54:17.958091 4040 raft_consensus.cc:2241] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:17.958524 4040 raft_consensus.cc:2270] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:17.959383 4037 tablet_bootstrap.cc:492] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:17.961670 4037 tablet_bootstrap.cc:654] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:17.965009 4040 ts_tablet_manager.cc:1905] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:17.971319 4038 tablet_bootstrap.cc:492] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:17.971520 4038 ts_tablet_manager.cc:1397] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.004s
I20250901 19:54:17.971904 4037 tablet_bootstrap.cc:492] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:17.972088 4037 ts_tablet_manager.cc:1397] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.001s
I20250901 19:54:17.972525 4038 raft_consensus.cc:357] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:17.972736 4038 raft_consensus.cc:383] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:17.972846 4038 raft_consensus.cc:738] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:17.972965 4037 raft_consensus.cc:357] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:17.973151 4037 raft_consensus.cc:383] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:17.973106 4038 consensus_queue.cc:260] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:17.973249 4037 raft_consensus.cc:738] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:17.973523 4037 consensus_queue.cc:260] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:17.974467 4038 ts_tablet_manager.cc:1428] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 19:54:17.975400 4038 tablet_bootstrap.cc:492] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:17.975445 4037 ts_tablet_manager.cc:1428] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250901 19:54:17.976269 4036 tablet_bootstrap.cc:492] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:17.976524 4036 ts_tablet_manager.cc:1397] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.004s
I20250901 19:54:17.977532 4038 tablet_bootstrap.cc:654] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:17.977708 4036 raft_consensus.cc:357] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:17.977943 4036 raft_consensus.cc:383] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:17.978027 4036 raft_consensus.cc:738] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:17.978386 4036 consensus_queue.cc:260] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:17.980242 4036 ts_tablet_manager.cc:1428] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250901 19:54:17.980681 4036 tablet_bootstrap.cc:492] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:17.983039 4036 tablet_bootstrap.cc:654] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:17.986595 4038 tablet_bootstrap.cc:492] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:17.986840 4038 ts_tablet_manager.cc:1397] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250901 19:54:17.986953 3758 tablet_service.cc:1468] Processing CreateTablet for tablet f4fe9cadf6414b548cbd95572d9ed1c8 (DEFAULT_TABLE table=default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.987399 3758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4fe9cadf6414b548cbd95572d9ed1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:17.987485 3617 tablet_service.cc:1468] Processing CreateTablet for tablet f4fe9cadf6414b548cbd95572d9ed1c8 (DEFAULT_TABLE table=default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.987730 4038 raft_consensus.cc:357] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:17.987901 3617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4fe9cadf6414b548cbd95572d9ed1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:17.987933 4038 raft_consensus.cc:383] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:17.988005 4038 raft_consensus.cc:738] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:17.988297 4038 consensus_queue.cc:260] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:17.991005 4038 ts_tablet_manager.cc:1428] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250901 19:54:17.995018 3895 tablet_service.cc:1468] Processing CreateTablet for tablet f4fe9cadf6414b548cbd95572d9ed1c8 (DEFAULT_TABLE table=default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:17.995525 3895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4fe9cadf6414b548cbd95572d9ed1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.008399 4038 tablet_bootstrap.cc:492] T f4fe9cadf6414b548cbd95572d9ed1c8 P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:18.009155 4037 tablet_bootstrap.cc:492] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:18.010247 4038 tablet_bootstrap.cc:654] T f4fe9cadf6414b548cbd95572d9ed1c8 P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.012589 3758 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a600536db3a40b28f031061d02a49a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d8339cbcc774ea5bae71abeb3ecd2b7) from {username='slave'} at 127.0.0.1:52658
I20250901 19:54:18.013073 4042 tablet_replica.cc:331] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734: stopping tablet replica
I20250901 19:54:18.013813 4042 raft_consensus.cc:2241] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:18.014400 4042 raft_consensus.cc:2270] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:18.014708 4037 tablet_bootstrap.cc:654] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.016620 4042 ts_tablet_manager.cc:1905] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:18.018276 4040 ts_tablet_manager.cc:1918] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 19:54:18.018631 4040 log.cc:1199] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/wal/wals/1a600536db3a40b28f031061d02a49a6
I20250901 19:54:18.019400 4040 ts_tablet_manager.cc:1939] T 1a600536db3a40b28f031061d02a49a6 P e0064a1b18e5470b85bde2277757aac8: Deleting consensus metadata
I20250901 19:54:18.022351 3293 catalog_manager.cc:4928] TS e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271): tablet 1a600536db3a40b28f031061d02a49a6 (table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]) successfully deleted
W20250901 19:54:18.023541 3831 tablet.cc:2378] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 19:54:18.023854 4037 tablet_bootstrap.cc:492] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:18.024042 4037 ts_tablet_manager.cc:1397] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.001s
I20250901 19:54:18.025045 4042 ts_tablet_manager.cc:1918] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 19:54:18.024942 4037 raft_consensus.cc:357] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.025198 4037 raft_consensus.cc:383] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.025300 4037 raft_consensus.cc:738] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.025592 4037 consensus_queue.cc:260] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.026196 4037 ts_tablet_manager.cc:1428] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 19:54:18.026512 4037 tablet_bootstrap.cc:492] T f4fe9cadf6414b548cbd95572d9ed1c8 P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:18.026923 4036 tablet_bootstrap.cc:492] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:18.027202 4036 ts_tablet_manager.cc:1397] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.015s
I20250901 19:54:18.028384 4037 tablet_bootstrap.cc:654] T f4fe9cadf6414b548cbd95572d9ed1c8 P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.028396 4036 raft_consensus.cc:357] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.028805 4036 raft_consensus.cc:383] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.028970 4036 raft_consensus.cc:738] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.029335 4036 consensus_queue.cc:260] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.033558 4046 raft_consensus.cc:491] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:18.033816 4046 raft_consensus.cc:513] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.036350 4046 leader_election.cc:290] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:18.037158 3617 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a600536db3a40b28f031061d02a49a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d8339cbcc774ea5bae71abeb3ecd2b7) from {username='slave'} at 127.0.0.1:43308
I20250901 19:54:18.037528 4041 tablet_replica.cc:331] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21: stopping tablet replica
I20250901 19:54:18.038021 4041 raft_consensus.cc:2241] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:18.038486 4041 raft_consensus.cc:2270] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:18.040596 4049 raft_consensus.cc:491] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:18.040876 4049 raft_consensus.cc:513] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.041270 4036 ts_tablet_manager.cc:1428] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250901 19:54:18.041698 4036 tablet_bootstrap.cc:492] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:18.043256 4049 leader_election.cc:290] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d397574faadb441192c99d530f14a734 (127.2.246.194:33557), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.025215 4042 log.cc:1199] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/wal/wals/1a600536db3a40b28f031061d02a49a6
I20250901 19:54:18.044203 4042 ts_tablet_manager.cc:1939] T 1a600536db3a40b28f031061d02a49a6 P d397574faadb441192c99d530f14a734: Deleting consensus metadata
W20250901 19:54:18.045903 3964 tablet.cc:2378] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 19:54:18.046217 3294 catalog_manager.cc:4928] TS d397574faadb441192c99d530f14a734 (127.2.246.194:33557): tablet 1a600536db3a40b28f031061d02a49a6 (table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]) successfully deleted
I20250901 19:54:18.047302 4036 tablet_bootstrap.cc:654] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.048556 3638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d8339cbcc774ea5bae71abeb3ecd2b7" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21" is_pre_election: true
I20250901 19:54:18.049826 3758 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a600536db3a40b28f031061d02a49a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d8339cbcc774ea5bae71abeb3ecd2b7) from {username='slave'} at 127.0.0.1:52658
W20250901 19:54:18.052171 3849 leader_election.cc:343] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Illegal state: must be running to vote when last-logged opid is not known
I20250901 19:54:18.053982 4041 ts_tablet_manager.cc:1905] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:18.075400 4037 tablet_bootstrap.cc:492] T f4fe9cadf6414b548cbd95572d9ed1c8 P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
W20250901 19:54:18.075424 3686 tablet.cc:2378] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 19:54:18.075610 4037 ts_tablet_manager.cc:1397] T f4fe9cadf6414b548cbd95572d9ed1c8 P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.003s
I20250901 19:54:18.076612 4037 raft_consensus.cc:357] T f4fe9cadf6414b548cbd95572d9ed1c8 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.076843 4037 raft_consensus.cc:383] T f4fe9cadf6414b548cbd95572d9ed1c8 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.076943 4037 raft_consensus.cc:738] T f4fe9cadf6414b548cbd95572d9ed1c8 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.077227 4037 consensus_queue.cc:260] T f4fe9cadf6414b548cbd95572d9ed1c8 P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.079375 4041 ts_tablet_manager.cc:1918] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 19:54:18.079608 4041 log.cc:1199] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/wal/wals/1a600536db3a40b28f031061d02a49a6
I20250901 19:54:18.079769 4037 ts_tablet_manager.cc:1428] T f4fe9cadf6414b548cbd95572d9ed1c8 P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250901 19:54:18.080183 4041 ts_tablet_manager.cc:1939] T 1a600536db3a40b28f031061d02a49a6 P c5b48d93671f4eb8980759b876e9ca21: Deleting consensus metadata
W20250901 19:54:18.081409 3294 catalog_manager.cc:4895] TS d397574faadb441192c99d530f14a734 (127.2.246.194:33557): delete failed for tablet 1a600536db3a40b28f031061d02a49a6 because the tablet was not found. No further retry: Not found: Tablet not found: 1a600536db3a40b28f031061d02a49a6
I20250901 19:54:18.082098 3292 catalog_manager.cc:4928] TS c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): tablet 1a600536db3a40b28f031061d02a49a6 (table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]) successfully deleted
I20250901 19:54:18.084455 3302 master_service.cc:948] ReplaceTablet: received request to replace tablet 1a600536db3a40b28f031061d02a49a6 from {username='slave'} at 127.0.0.1:55634
I20250901 19:54:18.085260 3302 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55634:
name: "default.Table_6f60f64babef49459553b6e490e198e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:18.085232 3300 catalog_manager.cc:2149] created HMS catalog entry for table [id=d9eb1bc0699747928b2a9d2e6fb8bbd8]
I20250901 19:54:18.090760 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d8339cbcc774ea5bae71abeb3ecd2b7" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734" is_pre_election: true
I20250901 19:54:18.091318 3778 raft_consensus.cc:2466] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 0.
I20250901 19:54:18.092289 3851 leader_election.cc:304] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d397574faadb441192c99d530f14a734, e0064a1b18e5470b85bde2277757aac8; no voters: c5b48d93671f4eb8980759b876e9ca21
I20250901 19:54:18.092824 4046 raft_consensus.cc:2802] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:18.092988 4046 raft_consensus.cc:491] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:18.093087 4046 raft_consensus.cc:3058] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.093271 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f640cc3c97394e16b88f7ef8148c19dd" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734" is_pre_election: true
I20250901 19:54:18.093477 3778 raft_consensus.cc:2466] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
I20250901 19:54:18.096050 4046 raft_consensus.cc:513] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.097538 3638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d8339cbcc774ea5bae71abeb3ecd2b7" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21"
I20250901 19:54:18.098037 3573 leader_election.cc:304] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters:
W20250901 19:54:18.098191 3849 leader_election.cc:343] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Illegal state: must be running to vote when last-logged opid is not known
I20250901 19:54:18.099164 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d8339cbcc774ea5bae71abeb3ecd2b7" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734"
I20250901 19:54:18.099378 3778 raft_consensus.cc:3058] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.102687 3778 raft_consensus.cc:2466] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 1.
I20250901 19:54:18.103749 3851 leader_election.cc:304] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d397574faadb441192c99d530f14a734, e0064a1b18e5470b85bde2277757aac8; no voters: c5b48d93671f4eb8980759b876e9ca21
I20250901 19:54:18.105453 4046 leader_election.cc:290] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 election: Requested vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:18.105978 4046 raft_consensus.cc:2802] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:18.106492 4046 raft_consensus.cc:695] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [term 1 LEADER]: Becoming Leader. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Running, Role: LEADER
I20250901 19:54:18.107237 4046 consensus_queue.cc:237] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.115278 3895 tablet_service.cc:1468] Processing CreateTablet for tablet 516ed4484b174a8d8cb8c6b9f076d31e (DEFAULT_TABLE table=default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.115954 3895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 516ed4484b174a8d8cb8c6b9f076d31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.120924 3617 tablet_service.cc:1468] Processing CreateTablet for tablet 516ed4484b174a8d8cb8c6b9f076d31e (DEFAULT_TABLE table=default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.121580 3617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 516ed4484b174a8d8cb8c6b9f076d31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.124663 3758 tablet_service.cc:1468] Processing CreateTablet for tablet 516ed4484b174a8d8cb8c6b9f076d31e (DEFAULT_TABLE table=default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.125154 3758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 516ed4484b174a8d8cb8c6b9f076d31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.127982 3299 catalog_manager.cc:2149] created HMS catalog entry for table [id=b2d5ec72c24e4c6f96918db0e2751612]
I20250901 19:54:18.128193 4049 raft_consensus.cc:2802] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:18.128387 4049 raft_consensus.cc:491] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:18.128494 4049 raft_consensus.cc:3058] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.132537 4049 raft_consensus.cc:513] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.133368 4049 leader_election.cc:290] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Requested vote from peers d397574faadb441192c99d530f14a734 (127.2.246.194:33557), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.134302 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f640cc3c97394e16b88f7ef8148c19dd" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734"
I20250901 19:54:18.134646 3778 raft_consensus.cc:3058] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.137300 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f640cc3c97394e16b88f7ef8148c19dd" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8" is_pre_election: true
I20250901 19:54:18.137720 3916 raft_consensus.cc:2466] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
I20250901 19:54:18.137821 3778 raft_consensus.cc:2466] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
I20250901 19:54:18.139268 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f640cc3c97394e16b88f7ef8148c19dd" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8"
I20250901 19:54:18.139467 3916 raft_consensus.cc:3058] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.139714 3573 leader_election.cc:304] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters:
I20250901 19:54:18.142598 3916 raft_consensus.cc:2466] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
I20250901 19:54:18.147550 4038 tablet_bootstrap.cc:492] T f4fe9cadf6414b548cbd95572d9ed1c8 P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:18.148386 4038 ts_tablet_manager.cc:1397] T f4fe9cadf6414b548cbd95572d9ed1c8 P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.140s user 0.007s sys 0.018s
I20250901 19:54:18.149345 4038 raft_consensus.cc:357] T f4fe9cadf6414b548cbd95572d9ed1c8 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.149580 4038 raft_consensus.cc:383] T f4fe9cadf6414b548cbd95572d9ed1c8 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.149679 4038 raft_consensus.cc:738] T f4fe9cadf6414b548cbd95572d9ed1c8 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.154755 4038 consensus_queue.cc:260] T f4fe9cadf6414b548cbd95572d9ed1c8 P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.156760 4038 ts_tablet_manager.cc:1428] T f4fe9cadf6414b548cbd95572d9ed1c8 P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250901 19:54:18.156219 3299 catalog_manager.cc:5582] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0064a1b18e5470b85bde2277757aac8 (127.2.246.195). New cstate: current_term: 1 leader_uuid: "e0064a1b18e5470b85bde2277757aac8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } health_report { overall_health: HEALTHY } } }
I20250901 19:54:18.159425 4036 tablet_bootstrap.cc:492] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:18.159667 4036 ts_tablet_manager.cc:1397] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.118s user 0.031s sys 0.041s
I20250901 19:54:18.160537 4036 raft_consensus.cc:357] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.160722 4036 raft_consensus.cc:383] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.160815 4036 raft_consensus.cc:738] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.161073 4036 consensus_queue.cc:260] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.167547 3300 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55634:
table { table_name: "default.Table_ea3686ff3ba94b9a984657e263ca0832" } new_table_name: "default.Table_f79692d68cd04e74ae3e3c67bc4f99a3" modify_external_catalogs: true
I20250901 19:54:18.168125 4063 raft_consensus.cc:491] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:18.168370 4063 raft_consensus.cc:513] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.169564 4063 leader_election.cc:290] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:18.169910 4049 raft_consensus.cc:2802] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:18.170615 3638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d21a96219ab4f5f96bc6d9976f990ff" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21" is_pre_election: true
I20250901 19:54:18.170958 3638 raft_consensus.cc:2466] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 0.
I20250901 19:54:18.171922 3849 leader_election.cc:304] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, e0064a1b18e5470b85bde2277757aac8; no voters:
I20250901 19:54:18.172209 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d21a96219ab4f5f96bc6d9976f990ff" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734" is_pre_election: true
I20250901 19:54:18.172395 4063 raft_consensus.cc:2802] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:18.172461 3778 raft_consensus.cc:2466] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 0.
I20250901 19:54:18.172544 4063 raft_consensus.cc:491] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:18.172665 4063 raft_consensus.cc:3058] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.173107 4038 tablet_bootstrap.cc:492] T 516ed4484b174a8d8cb8c6b9f076d31e P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:18.175210 4038 tablet_bootstrap.cc:654] T 516ed4484b174a8d8cb8c6b9f076d31e P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.175912 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.252:37003
--webserver_interface=127.2.246.252
--webserver_port=36357
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250901 19:54:18.177179 4037 tablet_bootstrap.cc:492] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:18.177875 4038 tablet_bootstrap.cc:492] T 516ed4484b174a8d8cb8c6b9f076d31e P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:18.178058 4038 ts_tablet_manager.cc:1397] T 516ed4484b174a8d8cb8c6b9f076d31e P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250901 19:54:18.178881 4038 raft_consensus.cc:357] T 516ed4484b174a8d8cb8c6b9f076d31e P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.179075 4037 tablet_bootstrap.cc:654] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.179126 4038 raft_consensus.cc:383] T 516ed4484b174a8d8cb8c6b9f076d31e P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.179319 4038 raft_consensus.cc:738] T 516ed4484b174a8d8cb8c6b9f076d31e P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.179735 4038 consensus_queue.cc:260] T 516ed4484b174a8d8cb8c6b9f076d31e P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.185173 4063 raft_consensus.cc:513] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.186702 3638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d21a96219ab4f5f96bc6d9976f990ff" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21"
I20250901 19:54:18.186985 3638 raft_consensus.cc:3058] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.188767 3617 tablet_service.cc:1468] Processing CreateTablet for tablet a5067dd498ef4331bd9d9f3bcbcaec44 (DEFAULT_TABLE table=default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.189316 3617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5067dd498ef4331bd9d9f3bcbcaec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.191330 4038 ts_tablet_manager.cc:1428] T 516ed4484b174a8d8cb8c6b9f076d31e P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20250901 19:54:18.192621 3638 raft_consensus.cc:2466] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 1.
I20250901 19:54:18.192625 3895 tablet_service.cc:1468] Processing CreateTablet for tablet a5067dd498ef4331bd9d9f3bcbcaec44 (DEFAULT_TABLE table=default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.193300 3758 tablet_service.cc:1468] Processing CreateTablet for tablet a5067dd498ef4331bd9d9f3bcbcaec44 (DEFAULT_TABLE table=default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.193329 3849 leader_election.cc:304] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, e0064a1b18e5470b85bde2277757aac8; no voters:
I20250901 19:54:18.193805 4046 raft_consensus.cc:2802] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:18.198673 3299 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55634:
name: "default.Table_b7d788fb008945558d1e6fd4dde4f706"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:18.199430 4063 leader_election.cc:290] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 election: Requested vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:18.204757 3895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5067dd498ef4331bd9d9f3bcbcaec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.199584 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d21a96219ab4f5f96bc6d9976f990ff" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734"
I20250901 19:54:18.208125 3778 raft_consensus.cc:3058] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.208377 3758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5067dd498ef4331bd9d9f3bcbcaec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.211827 3778 raft_consensus.cc:2466] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 1.
I20250901 19:54:18.224350 3303 catalog_manager.cc:2149] created HMS catalog entry for table [id=513fdaec16fd4268bf9ac4b9cd6cfd59]
I20250901 19:54:18.225895 4046 raft_consensus.cc:695] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [term 1 LEADER]: Becoming Leader. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Running, Role: LEADER
I20250901 19:54:18.227813 4037 tablet_bootstrap.cc:492] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:18.228082 4037 ts_tablet_manager.cc:1397] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.002s
I20250901 19:54:18.230720 4037 raft_consensus.cc:357] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.231032 4037 raft_consensus.cc:383] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.231145 4037 raft_consensus.cc:738] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.231590 4037 consensus_queue.cc:260] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.231699 3303 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55634:
name: "default.Table_b679bfff7a2f4d13b5e318b2ec42fc36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:18.226282 4046 consensus_queue.cc:237] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.242327 3896 tablet_service.cc:1468] Processing CreateTablet for tablet 5f37af30f3894824b7ce48101e1fda8a (DEFAULT_TABLE table=default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.242959 3896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f37af30f3894824b7ce48101e1fda8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.248690 3757 tablet_service.cc:1468] Processing CreateTablet for tablet 5f37af30f3894824b7ce48101e1fda8a (DEFAULT_TABLE table=default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.248989 3616 tablet_service.cc:1468] Processing CreateTablet for tablet 5f37af30f3894824b7ce48101e1fda8a (DEFAULT_TABLE table=default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.249226 3757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f37af30f3894824b7ce48101e1fda8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.249536 3616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f37af30f3894824b7ce48101e1fda8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.252624 4038 tablet_bootstrap.cc:492] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:18.258328 4049 raft_consensus.cc:695] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [term 1 LEADER]: Becoming Leader. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Running, Role: LEADER
I20250901 19:54:18.259291 4049 consensus_queue.cc:237] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.261446 4038 tablet_bootstrap.cc:654] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.272583 4036 ts_tablet_manager.cc:1428] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.113s user 0.033s sys 0.039s
I20250901 19:54:18.273029 4036 tablet_bootstrap.cc:492] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:18.274536 4038 tablet_bootstrap.cc:492] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:18.274708 4038 ts_tablet_manager.cc:1397] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.009s
I20250901 19:54:18.274755 4037 ts_tablet_manager.cc:1428] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.047s user 0.000s sys 0.003s
I20250901 19:54:18.275662 4038 raft_consensus.cc:357] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.275872 4038 raft_consensus.cc:383] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.275959 4038 raft_consensus.cc:738] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.276244 4038 consensus_queue.cc:260] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.277621 4037 tablet_bootstrap.cc:492] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:18.277698 4036 tablet_bootstrap.cc:654] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.279917 4037 tablet_bootstrap.cc:654] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.280293 4038 ts_tablet_manager.cc:1428] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250901 19:54:18.283571 4038 tablet_bootstrap.cc:492] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:18.285568 4038 tablet_bootstrap.cc:654] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.291356 4036 tablet_bootstrap.cc:492] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:18.291559 4036 ts_tablet_manager.cc:1397] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.004s
I20250901 19:54:18.292589 4036 raft_consensus.cc:357] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.292786 4036 raft_consensus.cc:383] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.292884 4036 raft_consensus.cc:738] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.293164 4036 consensus_queue.cc:260] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.296934 4036 ts_tablet_manager.cc:1428] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250901 19:54:18.300163 4036 tablet_bootstrap.cc:492] T 516ed4484b174a8d8cb8c6b9f076d31e P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:18.300663 4037 tablet_bootstrap.cc:492] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:18.300891 4037 ts_tablet_manager.cc:1397] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.003s
I20250901 19:54:18.301846 4037 raft_consensus.cc:357] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.302112 4037 raft_consensus.cc:383] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.302197 4036 tablet_bootstrap.cc:654] T 516ed4484b174a8d8cb8c6b9f076d31e P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.302304 4037 raft_consensus.cc:738] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.302634 4037 consensus_queue.cc:260] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.305024 4037 ts_tablet_manager.cc:1428] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250901 19:54:18.306788 4037 tablet_bootstrap.cc:492] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:18.309311 4037 tablet_bootstrap.cc:654] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.310225 4036 tablet_bootstrap.cc:492] T 516ed4484b174a8d8cb8c6b9f076d31e P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:18.310406 4036 ts_tablet_manager.cc:1397] T 516ed4484b174a8d8cb8c6b9f076d31e P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20250901 19:54:18.311280 4036 raft_consensus.cc:357] T 516ed4484b174a8d8cb8c6b9f076d31e P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.311489 4036 raft_consensus.cc:383] T 516ed4484b174a8d8cb8c6b9f076d31e P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.311553 4036 raft_consensus.cc:738] T 516ed4484b174a8d8cb8c6b9f076d31e P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.311895 4036 consensus_queue.cc:260] T 516ed4484b174a8d8cb8c6b9f076d31e P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.315112 4036 ts_tablet_manager.cc:1428] T 516ed4484b174a8d8cb8c6b9f076d31e P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 19:54:18.317691 4036 tablet_bootstrap.cc:492] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:18.319525 4036 tablet_bootstrap.cc:654] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.328069 4036 tablet_bootstrap.cc:492] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:18.328298 4036 ts_tablet_manager.cc:1397] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20250901 19:54:18.329492 4036 raft_consensus.cc:357] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.329756 4036 raft_consensus.cc:383] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.329839 4036 raft_consensus.cc:738] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.330196 4036 consensus_queue.cc:260] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.334256 3301 catalog_manager.cc:2149] created HMS catalog entry for table [id=78b1dcafd0d04873a71930d424cf2d67]
I20250901 19:54:18.334625 4036 ts_tablet_manager.cc:1428] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250901 19:54:18.337766 4036 tablet_bootstrap.cc:492] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:18.339843 4036 tablet_bootstrap.cc:654] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.345242 4037 tablet_bootstrap.cc:492] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:18.345515 4037 ts_tablet_manager.cc:1397] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.003s
I20250901 19:54:18.346407 4037 raft_consensus.cc:357] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.346637 4037 raft_consensus.cc:383] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.346714 4037 raft_consensus.cc:738] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.347061 4037 consensus_queue.cc:260] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.348832 4037 ts_tablet_manager.cc:1428] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 19:54:18.351439 4036 tablet_bootstrap.cc:492] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:18.351619 4036 ts_tablet_manager.cc:1397] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20250901 19:54:18.352476 4036 raft_consensus.cc:357] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.352679 4036 raft_consensus.cc:383] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.352744 4036 raft_consensus.cc:738] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.353050 4036 consensus_queue.cc:260] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.357180 4036 ts_tablet_manager.cc:1428] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 19:54:18.360884 3617 tablet_service.cc:1468] Processing CreateTablet for tablet e43433b1ce9646e49962fe44bbed2491 (DEFAULT_TABLE table=default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.361436 3617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e43433b1ce9646e49962fe44bbed2491. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.368065 3895 tablet_service.cc:1468] Processing CreateTablet for tablet e43433b1ce9646e49962fe44bbed2491 (DEFAULT_TABLE table=default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.368623 3895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e43433b1ce9646e49962fe44bbed2491. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.370105 3301 catalog_manager.cc:5582] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0064a1b18e5470b85bde2277757aac8 (127.2.246.195). New cstate: current_term: 1 leader_uuid: "e0064a1b18e5470b85bde2277757aac8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } health_report { overall_health: HEALTHY } } }
I20250901 19:54:18.371585 4036 tablet_bootstrap.cc:492] T e43433b1ce9646e49962fe44bbed2491 P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:18.371573 3758 tablet_service.cc:1468] Processing CreateTablet for tablet e43433b1ce9646e49962fe44bbed2491 (DEFAULT_TABLE table=default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.372092 3758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e43433b1ce9646e49962fe44bbed2491. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.373586 4036 tablet_bootstrap.cc:654] T e43433b1ce9646e49962fe44bbed2491 P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.373809 4038 tablet_bootstrap.cc:492] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:18.374008 4038 ts_tablet_manager.cc:1397] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.091s user 0.017s sys 0.020s
I20250901 19:54:18.375003 4038 raft_consensus.cc:357] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.375275 4038 raft_consensus.cc:383] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.375361 4038 raft_consensus.cc:738] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.378612 4038 consensus_queue.cc:260] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.379231 4038 ts_tablet_manager.cc:1428] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 19:54:18.382562 4037 tablet_bootstrap.cc:492] T e43433b1ce9646e49962fe44bbed2491 P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:18.384450 4037 tablet_bootstrap.cc:654] T e43433b1ce9646e49962fe44bbed2491 P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.389055 4038 tablet_bootstrap.cc:492] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:18.390967 4038 tablet_bootstrap.cc:654] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.394924 3301 catalog_manager.cc:5582] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193). New cstate: current_term: 1 leader_uuid: "c5b48d93671f4eb8980759b876e9ca21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20250901 19:54:18.398041 4036 tablet_bootstrap.cc:492] T e43433b1ce9646e49962fe44bbed2491 P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:18.398226 4036 ts_tablet_manager.cc:1397] T e43433b1ce9646e49962fe44bbed2491 P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.006s
I20250901 19:54:18.399113 4036 raft_consensus.cc:357] T e43433b1ce9646e49962fe44bbed2491 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.399315 4036 raft_consensus.cc:383] T e43433b1ce9646e49962fe44bbed2491 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.399436 4036 raft_consensus.cc:738] T e43433b1ce9646e49962fe44bbed2491 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.399749 4036 consensus_queue.cc:260] T e43433b1ce9646e49962fe44bbed2491 P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.401623 4036 ts_tablet_manager.cc:1428] T e43433b1ce9646e49962fe44bbed2491 P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250901 19:54:18.405272 4038 tablet_bootstrap.cc:492] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:18.405467 4038 ts_tablet_manager.cc:1397] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20250901 19:54:18.406349 4038 raft_consensus.cc:357] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.408497 4038 raft_consensus.cc:383] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.408676 4038 raft_consensus.cc:738] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.408972 4038 consensus_queue.cc:260] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.409638 4038 ts_tablet_manager.cc:1428] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250901 19:54:18.426662 4037 tablet_bootstrap.cc:492] T e43433b1ce9646e49962fe44bbed2491 P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:18.426950 4037 ts_tablet_manager.cc:1397] T e43433b1ce9646e49962fe44bbed2491 P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.016s
I20250901 19:54:18.428669 4037 raft_consensus.cc:357] T e43433b1ce9646e49962fe44bbed2491 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.429041 4037 raft_consensus.cc:383] T e43433b1ce9646e49962fe44bbed2491 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.430756 3301 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55634:
table { table_name: "default.Table_69ed2a785efc4a4f8077c8aa2ef89c35" } new_table_name: "default.Table_6d9fb4a3f26048cb95788236b2d5ec7f" modify_external_catalogs: true
I20250901 19:54:18.431195 4037 raft_consensus.cc:738] T e43433b1ce9646e49962fe44bbed2491 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.431491 4037 consensus_queue.cc:260] T e43433b1ce9646e49962fe44bbed2491 P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.432078 4037 ts_tablet_manager.cc:1428] T e43433b1ce9646e49962fe44bbed2491 P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 19:54:18.434485 3305 catalog_manager.cc:2149] created HMS catalog entry for table [id=2c6ea8bb63684f33986a43cd87476077]
I20250901 19:54:18.441517 3305 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55634 (request call id 10) took 1011 ms. Trace:
I20250901 19:54:18.442977 4047 raft_consensus.cc:491] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:18.441707 3305 rpcz_store.cc:276] 0901 19:54:17.429684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 19:54:17.430193 (+ 509us) service_pool.cc:225] Handling call
0901 19:54:17.453874 (+ 23681us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 19:54:17.454302 (+ 428us) catalog_manager.cc:2122] Created new table and tablet info
0901 19:54:18.434430 (+980128us) catalog_manager.cc:2148] Created new table in HMS catalog
0901 19:54:18.441206 (+ 6776us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 19:54:18.441345 (+ 139us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 19:54:18.441467 (+ 122us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 19:54:18.435317 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 19:54:18.435447 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0901 19:54:18.435457 (+ 10us) write_op.cc:435] Acquired schema lock
0901 19:54:18.435460 (+ 3us) tablet.cc:660] Decoding operations
0901 19:54:18.435564 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0901 19:54:18.435576 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0901 19:54:18.435579 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0901 19:54:18.435683 (+ 104us) tablet.cc:699] Row locks acquired
0901 19:54:18.435686 (+ 3us) write_op.cc:260] PREPARE: finished
0901 19:54:18.435732 (+ 46us) write_op.cc:270] Start()
0901 19:54:18.435768 (+ 36us) write_op.cc:276] Timestamp: P: 1756756458435701 usec, L: 0
0901 19:54:18.435771 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 19:54:18.435956 (+ 185us) log.cc:844] Serialized 432 byte log entry
0901 19:54:18.439477 (+ 3521us) op_driver.cc:464] REPLICATION: finished
0901 19:54:18.439660 (+ 183us) write_op.cc:301] APPLY: starting
0901 19:54:18.439680 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0901 19:54:18.439730 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0901 19:54:18.439771 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 19:54:18.439883 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 19:54:18.439921 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 19:54:18.439929 (+ 8us) write_op.cc:312] APPLY: finished
0901 19:54:18.440032 (+ 103us) log.cc:844] Serialized 28 byte log entry
0901 19:54:18.440105 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0901 19:54:18.440129 (+ 24us) write_op.cc:454] Released schema lock
0901 19:54:18.440136 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 19:54:18.440229 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0901 19:54:18.440234 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":879767,"Hive Metastore.run_cpu_time_us":314,"Hive Metastore.run_wall_time_us":99803,"child_traces":[["op",{"apply.queue_time_us":96,"apply.run_cpu_time_us":588,"apply.run_wall_time_us":588,"num_ops":2,"prepare.queue_time_us":76,"prepare.run_cpu_time_us":782,"prepare.run_wall_time_us":781,"raft.queue_time_us":200,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":329,"replication_time_us":3671}]]}
I20250901 19:54:18.443176 4047 raft_consensus.cc:513] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.445539 4047 leader_election.cc:290] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.452828 3895 tablet_service.cc:1468] Processing CreateTablet for tablet 7ab5ab06eece48fdb346f232247f72e0 (DEFAULT_TABLE table=default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.453389 3895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab5ab06eece48fdb346f232247f72e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.456312 3617 tablet_service.cc:1468] Processing CreateTablet for tablet 7ab5ab06eece48fdb346f232247f72e0 (DEFAULT_TABLE table=default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.456864 3617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab5ab06eece48fdb346f232247f72e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.459311 3305 master_service.cc:948] ReplaceTablet: received request to replace tablet f640cc3c97394e16b88f7ef8148c19dd from {username='slave'} at 127.0.0.1:55634
I20250901 19:54:18.460785 3758 tablet_service.cc:1468] Processing CreateTablet for tablet 7ab5ab06eece48fdb346f232247f72e0 (DEFAULT_TABLE table=default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.461257 3758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab5ab06eece48fdb346f232247f72e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.463784 4038 tablet_bootstrap.cc:492] T 7ab5ab06eece48fdb346f232247f72e0 P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:18.464325 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "516ed4484b174a8d8cb8c6b9f076d31e" candidate_uuid: "d397574faadb441192c99d530f14a734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8" is_pre_election: true
I20250901 19:54:18.464624 3916 raft_consensus.cc:2466] T 516ed4484b174a8d8cb8c6b9f076d31e P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d397574faadb441192c99d530f14a734 in term 0.
I20250901 19:54:18.465443 3712 leader_election.cc:304] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d397574faadb441192c99d530f14a734, e0064a1b18e5470b85bde2277757aac8; no voters:
I20250901 19:54:18.465691 4038 tablet_bootstrap.cc:654] T 7ab5ab06eece48fdb346f232247f72e0 P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.465893 4047 raft_consensus.cc:2802] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:18.466038 4047 raft_consensus.cc:491] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:18.466161 4047 raft_consensus.cc:3058] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.468639 3305 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Sending DeleteTablet for 3 replicas of tablet f640cc3c97394e16b88f7ef8148c19dd
I20250901 19:54:18.469560 3616 tablet_service.cc:1515] Processing DeleteTablet for tablet f640cc3c97394e16b88f7ef8148c19dd with delete_type TABLET_DATA_DELETED (replaced by tablet e1d516f8388c4a099f81d6e5a3c7cf4d) from {username='slave'} at 127.0.0.1:43308
I20250901 19:54:18.469594 3638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "516ed4484b174a8d8cb8c6b9f076d31e" candidate_uuid: "d397574faadb441192c99d530f14a734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21" is_pre_election: true
I20250901 19:54:18.469834 3305 catalog_manager.cc:6652] ReplaceTablet: tablet f640cc3c97394e16b88f7ef8148c19dd deleted and replaced by tablet e1d516f8388c4a099f81d6e5a3c7cf4d
I20250901 19:54:18.469878 3638 raft_consensus.cc:2466] T 516ed4484b174a8d8cb8c6b9f076d31e P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d397574faadb441192c99d530f14a734 in term 0.
I20250901 19:54:18.469878 4041 tablet_replica.cc:331] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21: stopping tablet replica
I20250901 19:54:18.470386 4041 raft_consensus.cc:2241] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [term 1 LEADER]: Raft consensus shutting down.
I20250901 19:54:18.470104 4047 raft_consensus.cc:513] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.470896 4041 pending_rounds.cc:70] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21: Trying to abort 1 pending ops.
I20250901 19:54:18.471158 4041 pending_rounds.cc:77] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195674453069221888 op_type: NO_OP noop_request { }
I20250901 19:54:18.471349 4041 raft_consensus.cc:2887] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 19:54:18.471509 4041 raft_consensus.cc:2270] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:18.471978 4047 leader_election.cc:290] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 election: Requested vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.472767 3638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "516ed4484b174a8d8cb8c6b9f076d31e" candidate_uuid: "d397574faadb441192c99d530f14a734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21"
I20250901 19:54:18.472988 3638 raft_consensus.cc:3058] T 516ed4484b174a8d8cb8c6b9f076d31e P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.476722 4041 ts_tablet_manager.cc:1905] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:18.469420 4038 tablet_bootstrap.cc:492] T 7ab5ab06eece48fdb346f232247f72e0 P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:18.477089 4038 ts_tablet_manager.cc:1397] T 7ab5ab06eece48fdb346f232247f72e0 P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20250901 19:54:18.477974 3757 tablet_service.cc:1515] Processing DeleteTablet for tablet f640cc3c97394e16b88f7ef8148c19dd with delete_type TABLET_DATA_DELETED (replaced by tablet e1d516f8388c4a099f81d6e5a3c7cf4d) from {username='slave'} at 127.0.0.1:52658
I20250901 19:54:18.477921 4038 raft_consensus.cc:357] T 7ab5ab06eece48fdb346f232247f72e0 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.478116 4038 raft_consensus.cc:383] T 7ab5ab06eece48fdb346f232247f72e0 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.478186 4038 raft_consensus.cc:738] T 7ab5ab06eece48fdb346f232247f72e0 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.478261 4042 tablet_replica.cc:331] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734: stopping tablet replica
I20250901 19:54:18.478442 4038 consensus_queue.cc:260] T 7ab5ab06eece48fdb346f232247f72e0 P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.478516 4042 raft_consensus.cc:2241] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:18.478703 4042 raft_consensus.cc:2270] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:18.479300 4042 ts_tablet_manager.cc:1905] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:18.480257 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "516ed4484b174a8d8cb8c6b9f076d31e" candidate_uuid: "d397574faadb441192c99d530f14a734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8"
I20250901 19:54:18.480451 3916 raft_consensus.cc:3058] T 516ed4484b174a8d8cb8c6b9f076d31e P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.481895 4038 ts_tablet_manager.cc:1428] T 7ab5ab06eece48fdb346f232247f72e0 P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250901 19:54:18.483245 3895 tablet_service.cc:1515] Processing DeleteTablet for tablet f640cc3c97394e16b88f7ef8148c19dd with delete_type TABLET_DATA_DELETED (replaced by tablet e1d516f8388c4a099f81d6e5a3c7cf4d) from {username='slave'} at 127.0.0.1:38840
I20250901 19:54:18.483312 3916 raft_consensus.cc:2466] T 516ed4484b174a8d8cb8c6b9f076d31e P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d397574faadb441192c99d530f14a734 in term 1.
I20250901 19:54:18.483606 4040 tablet_replica.cc:331] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8: stopping tablet replica
I20250901 19:54:18.483888 4040 raft_consensus.cc:2241] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:18.484057 4040 raft_consensus.cc:2270] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:18.484625 4040 ts_tablet_manager.cc:1905] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:18.484845 3712 leader_election.cc:304] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d397574faadb441192c99d530f14a734, e0064a1b18e5470b85bde2277757aac8; no voters:
I20250901 19:54:18.485965 4042 ts_tablet_manager.cc:1918] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 19:54:18.486117 4042 log.cc:1199] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/wal/wals/f640cc3c97394e16b88f7ef8148c19dd
I20250901 19:54:18.486479 4042 ts_tablet_manager.cc:1939] T f640cc3c97394e16b88f7ef8148c19dd P d397574faadb441192c99d530f14a734: Deleting consensus metadata
I20250901 19:54:18.486718 3638 raft_consensus.cc:2466] T 516ed4484b174a8d8cb8c6b9f076d31e P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d397574faadb441192c99d530f14a734 in term 1.
I20250901 19:54:18.488459 3294 catalog_manager.cc:4928] TS d397574faadb441192c99d530f14a734 (127.2.246.194:33557): tablet f640cc3c97394e16b88f7ef8148c19dd (table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]) successfully deleted
I20250901 19:54:18.488765 4047 raft_consensus.cc:2802] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:18.491075 4037 tablet_bootstrap.cc:492] T 7ab5ab06eece48fdb346f232247f72e0 P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:18.493216 4037 tablet_bootstrap.cc:654] T 7ab5ab06eece48fdb346f232247f72e0 P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.493897 3757 tablet_service.cc:1468] Processing CreateTablet for tablet e1d516f8388c4a099f81d6e5a3c7cf4d (DEFAULT_TABLE table=default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.493969 3895 tablet_service.cc:1468] Processing CreateTablet for tablet e1d516f8388c4a099f81d6e5a3c7cf4d (DEFAULT_TABLE table=default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.494369 3757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d516f8388c4a099f81d6e5a3c7cf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.494400 3895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d516f8388c4a099f81d6e5a3c7cf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.495272 3616 tablet_service.cc:1468] Processing CreateTablet for tablet e1d516f8388c4a099f81d6e5a3c7cf4d (DEFAULT_TABLE table=default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.495718 3616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d516f8388c4a099f81d6e5a3c7cf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.498842 4036 tablet_bootstrap.cc:492] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:18.501113 4036 tablet_bootstrap.cc:654] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.501653 4038 tablet_bootstrap.cc:492] T e1d516f8388c4a099f81d6e5a3c7cf4d P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:18.503947 4038 tablet_bootstrap.cc:654] T e1d516f8388c4a099f81d6e5a3c7cf4d P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.503872 4049 raft_consensus.cc:491] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:18.504149 4049 raft_consensus.cc:513] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.505033 4049 leader_election.cc:290] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d397574faadb441192c99d530f14a734 (127.2.246.194:33557), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.506345 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4fe9cadf6414b548cbd95572d9ed1c8" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734" is_pre_election: true
I20250901 19:54:18.506635 3778 raft_consensus.cc:2466] T f4fe9cadf6414b548cbd95572d9ed1c8 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
I20250901 19:54:18.507252 3573 leader_election.cc:304] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters:
I20250901 19:54:18.507710 4049 raft_consensus.cc:2802] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:18.507841 4049 raft_consensus.cc:491] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:18.507959 4049 raft_consensus.cc:3058] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.511041 4049 raft_consensus.cc:513] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.512215 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4fe9cadf6414b548cbd95572d9ed1c8" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8" is_pre_election: true
I20250901 19:54:18.512573 3916 raft_consensus.cc:2466] T f4fe9cadf6414b548cbd95572d9ed1c8 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
I20250901 19:54:18.514209 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4fe9cadf6414b548cbd95572d9ed1c8" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734"
I20250901 19:54:18.514460 3778 raft_consensus.cc:3058] T f4fe9cadf6414b548cbd95572d9ed1c8 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.515302 4040 ts_tablet_manager.cc:1918] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 19:54:18.515470 4040 log.cc:1199] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/wal/wals/f640cc3c97394e16b88f7ef8148c19dd
I20250901 19:54:18.516248 4038 tablet_bootstrap.cc:492] T e1d516f8388c4a099f81d6e5a3c7cf4d P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:18.516438 4038 ts_tablet_manager.cc:1397] T e1d516f8388c4a099f81d6e5a3c7cf4d P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20250901 19:54:18.517434 3778 raft_consensus.cc:2466] T f4fe9cadf6414b548cbd95572d9ed1c8 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
I20250901 19:54:18.517622 4038 raft_consensus.cc:357] T e1d516f8388c4a099f81d6e5a3c7cf4d P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.517889 4038 raft_consensus.cc:383] T e1d516f8388c4a099f81d6e5a3c7cf4d P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.518006 4038 raft_consensus.cc:738] T e1d516f8388c4a099f81d6e5a3c7cf4d P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.518224 3573 leader_election.cc:304] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters:
I20250901 19:54:18.518324 4038 consensus_queue.cc:260] T e1d516f8388c4a099f81d6e5a3c7cf4d P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.518635 4065 raft_consensus.cc:2802] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:18.518805 4065 raft_consensus.cc:695] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [term 1 LEADER]: Becoming Leader. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Running, Role: LEADER
I20250901 19:54:18.519122 4065 consensus_queue.cc:237] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.520890 4041 ts_tablet_manager.cc:1918] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 19:54:18.521059 4041 log.cc:1199] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/wal/wals/f640cc3c97394e16b88f7ef8148c19dd
I20250901 19:54:18.523608 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4fe9cadf6414b548cbd95572d9ed1c8" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8"
I20250901 19:54:18.523977 4038 ts_tablet_manager.cc:1428] T e1d516f8388c4a099f81d6e5a3c7cf4d P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20250901 19:54:18.524001 3916 raft_consensus.cc:3058] T f4fe9cadf6414b548cbd95572d9ed1c8 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.527501 3916 raft_consensus.cc:2466] T f4fe9cadf6414b548cbd95572d9ed1c8 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
I20250901 19:54:18.527920 4049 leader_election.cc:290] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Requested vote from peers d397574faadb441192c99d530f14a734 (127.2.246.194:33557), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.530097 4040 ts_tablet_manager.cc:1939] T f640cc3c97394e16b88f7ef8148c19dd P e0064a1b18e5470b85bde2277757aac8: Deleting consensus metadata
I20250901 19:54:18.531209 3293 catalog_manager.cc:4928] TS e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271): tablet f640cc3c97394e16b88f7ef8148c19dd (table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]) successfully deleted
I20250901 19:54:18.535681 4047 raft_consensus.cc:695] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [term 1 LEADER]: Becoming Leader. State: Replica: d397574faadb441192c99d530f14a734, State: Running, Role: LEADER
I20250901 19:54:18.535750 3305 catalog_manager.cc:5582] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193). New cstate: current_term: 1 leader_uuid: "c5b48d93671f4eb8980759b876e9ca21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20250901 19:54:18.536521 4047 consensus_queue.cc:237] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.540019 4041 ts_tablet_manager.cc:1939] T f640cc3c97394e16b88f7ef8148c19dd P c5b48d93671f4eb8980759b876e9ca21: Deleting consensus metadata
I20250901 19:54:18.544229 4037 tablet_bootstrap.cc:492] T 7ab5ab06eece48fdb346f232247f72e0 P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:18.544719 4037 ts_tablet_manager.cc:1397] T 7ab5ab06eece48fdb346f232247f72e0 P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.005s
I20250901 19:54:18.545595 3292 catalog_manager.cc:4928] TS c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): tablet f640cc3c97394e16b88f7ef8148c19dd (table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]) successfully deleted
I20250901 19:54:18.546808 4037 raft_consensus.cc:357] T 7ab5ab06eece48fdb346f232247f72e0 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.548079 4037 raft_consensus.cc:383] T 7ab5ab06eece48fdb346f232247f72e0 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.549952 4037 raft_consensus.cc:738] T 7ab5ab06eece48fdb346f232247f72e0 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.550534 4037 consensus_queue.cc:260] T 7ab5ab06eece48fdb346f232247f72e0 P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.551877 4036 tablet_bootstrap.cc:492] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:18.552177 4036 ts_tablet_manager.cc:1397] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.002s
I20250901 19:54:18.553176 4036 raft_consensus.cc:357] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.553517 4036 raft_consensus.cc:383] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.553647 4036 raft_consensus.cc:738] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.553567 3305 catalog_manager.cc:5582] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 reported cstate change: term changed from 0 to 1, leader changed from <none> to d397574faadb441192c99d530f14a734 (127.2.246.194). New cstate: current_term: 1 leader_uuid: "d397574faadb441192c99d530f14a734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20250901 19:54:18.554004 4036 consensus_queue.cc:260] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.554919 4036 ts_tablet_manager.cc:1428] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 19:54:18.557148 4037 ts_tablet_manager.cc:1428] T 7ab5ab06eece48fdb346f232247f72e0 P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250901 19:54:18.557593 4037 tablet_bootstrap.cc:492] T e1d516f8388c4a099f81d6e5a3c7cf4d P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:18.558542 4036 tablet_bootstrap.cc:492] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:18.564500 4065 raft_consensus.cc:491] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:18.564734 4065 raft_consensus.cc:513] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.565472 4065 leader_election.cc:290] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d397574faadb441192c99d530f14a734 (127.2.246.194:33557), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.566375 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ab5ab06eece48fdb346f232247f72e0" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734" is_pre_election: true
I20250901 19:54:18.566676 3778 raft_consensus.cc:2466] T 7ab5ab06eece48fdb346f232247f72e0 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
I20250901 19:54:18.567368 3573 leader_election.cc:304] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters:
I20250901 19:54:18.567674 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ab5ab06eece48fdb346f232247f72e0" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8" is_pre_election: true
I20250901 19:54:18.567835 4065 raft_consensus.cc:2802] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:18.567888 4036 tablet_bootstrap.cc:654] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.567950 4065 raft_consensus.cc:491] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:18.567970 3916 raft_consensus.cc:2466] T 7ab5ab06eece48fdb346f232247f72e0 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
I20250901 19:54:18.568049 4065 raft_consensus.cc:3058] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.570031 4037 tablet_bootstrap.cc:654] T e1d516f8388c4a099f81d6e5a3c7cf4d P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.572460 4065 raft_consensus.cc:513] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.573289 4065 leader_election.cc:290] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Requested vote from peers d397574faadb441192c99d530f14a734 (127.2.246.194:33557), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.573479 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ab5ab06eece48fdb346f232247f72e0" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734"
I20250901 19:54:18.573681 3778 raft_consensus.cc:3058] T 7ab5ab06eece48fdb346f232247f72e0 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.574980 3305 master_service.cc:948] ReplaceTablet: received request to replace tablet f640cc3c97394e16b88f7ef8148c19dd from {username='slave'} at 127.0.0.1:55634
I20250901 19:54:18.575873 3305 master_service.cc:948] ReplaceTablet: received request to replace tablet 0d8339cbcc774ea5bae71abeb3ecd2b7 from {username='slave'} at 127.0.0.1:55634
I20250901 19:54:18.576763 3778 raft_consensus.cc:2466] T 7ab5ab06eece48fdb346f232247f72e0 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
I20250901 19:54:18.578003 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ab5ab06eece48fdb346f232247f72e0" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8"
I20250901 19:54:18.578254 3916 raft_consensus.cc:3058] T 7ab5ab06eece48fdb346f232247f72e0 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.580175 3573 leader_election.cc:304] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters:
I20250901 19:54:18.580619 4065 raft_consensus.cc:2802] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:18.580786 4065 raft_consensus.cc:695] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [term 1 LEADER]: Becoming Leader. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Running, Role: LEADER
I20250901 19:54:18.581076 4065 consensus_queue.cc:237] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.583724 4036 tablet_bootstrap.cc:492] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:18.583904 4036 ts_tablet_manager.cc:1397] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.005s
I20250901 19:54:18.584757 4036 raft_consensus.cc:357] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.584981 4036 raft_consensus.cc:383] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.585099 4036 raft_consensus.cc:738] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.585352 4036 consensus_queue.cc:260] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.585914 3916 raft_consensus.cc:2466] T 7ab5ab06eece48fdb346f232247f72e0 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
I20250901 19:54:18.586593 3305 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Sending DeleteTablet for 3 replicas of tablet 0d8339cbcc774ea5bae71abeb3ecd2b7
I20250901 19:54:18.587414 3616 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d8339cbcc774ea5bae71abeb3ecd2b7 with delete_type TABLET_DATA_DELETED (replaced by tablet b867925c426149c99ffdd6d5df396462) from {username='slave'} at 127.0.0.1:43308
I20250901 19:54:18.587685 4041 tablet_replica.cc:331] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21: stopping tablet replica
I20250901 19:54:18.587895 4041 raft_consensus.cc:2241] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:18.588068 3757 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d8339cbcc774ea5bae71abeb3ecd2b7 with delete_type TABLET_DATA_DELETED (replaced by tablet b867925c426149c99ffdd6d5df396462) from {username='slave'} at 127.0.0.1:52658
I20250901 19:54:18.588369 4042 tablet_replica.cc:331] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734: stopping tablet replica
I20250901 19:54:18.588369 4041 raft_consensus.cc:2270] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:18.588563 4042 raft_consensus.cc:2241] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:18.588752 4042 raft_consensus.cc:2270] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:18.589223 4041 ts_tablet_manager.cc:1905] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:18.589265 4042 ts_tablet_manager.cc:1905] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:18.596419 4041 ts_tablet_manager.cc:1918] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 19:54:18.596431 4042 ts_tablet_manager.cc:1918] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 19:54:18.596572 4042 log.cc:1199] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/wal/wals/0d8339cbcc774ea5bae71abeb3ecd2b7
I20250901 19:54:18.596740 4041 log.cc:1199] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/wal/wals/0d8339cbcc774ea5bae71abeb3ecd2b7
I20250901 19:54:18.596978 4042 ts_tablet_manager.cc:1939] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P d397574faadb441192c99d530f14a734: Deleting consensus metadata
I20250901 19:54:18.597224 4041 ts_tablet_manager.cc:1939] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P c5b48d93671f4eb8980759b876e9ca21: Deleting consensus metadata
I20250901 19:54:18.598244 3292 catalog_manager.cc:4928] TS c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): tablet 0d8339cbcc774ea5bae71abeb3ecd2b7 (table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]) successfully deleted
I20250901 19:54:18.598552 4036 ts_tablet_manager.cc:1428] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250901 19:54:18.598604 3294 catalog_manager.cc:4928] TS d397574faadb441192c99d530f14a734 (127.2.246.194:33557): tablet 0d8339cbcc774ea5bae71abeb3ecd2b7 (table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]) successfully deleted
I20250901 19:54:18.599604 3895 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d8339cbcc774ea5bae71abeb3ecd2b7 with delete_type TABLET_DATA_DELETED (replaced by tablet b867925c426149c99ffdd6d5df396462) from {username='slave'} at 127.0.0.1:38840
I20250901 19:54:18.599934 4040 tablet_replica.cc:331] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8: stopping tablet replica
I20250901 19:54:18.600121 4040 raft_consensus.cc:2241] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [term 1 LEADER]: Raft consensus shutting down.
I20250901 19:54:18.600651 4040 pending_rounds.cc:70] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8: Trying to abort 1 pending ops.
I20250901 19:54:18.600828 4040 pending_rounds.cc:77] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195674452420591616 op_type: NO_OP noop_request { }
I20250901 19:54:18.601027 4040 raft_consensus.cc:2887] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 19:54:18.601145 4040 raft_consensus.cc:2270] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:18.602313 4040 ts_tablet_manager.cc:1905] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:18.602574 4037 tablet_bootstrap.cc:492] T e1d516f8388c4a099f81d6e5a3c7cf4d P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:18.602727 4037 ts_tablet_manager.cc:1397] T e1d516f8388c4a099f81d6e5a3c7cf4d P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.003s
I20250901 19:54:18.603587 4037 raft_consensus.cc:357] T e1d516f8388c4a099f81d6e5a3c7cf4d P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.603821 4037 raft_consensus.cc:383] T e1d516f8388c4a099f81d6e5a3c7cf4d P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.603931 4037 raft_consensus.cc:738] T e1d516f8388c4a099f81d6e5a3c7cf4d P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.604220 4037 consensus_queue.cc:260] T e1d516f8388c4a099f81d6e5a3c7cf4d P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.604777 4037 ts_tablet_manager.cc:1428] T e1d516f8388c4a099f81d6e5a3c7cf4d P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 19:54:18.609782 4040 ts_tablet_manager.cc:1918] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 19:54:18.611327 3895 tablet_service.cc:1468] Processing CreateTablet for tablet b867925c426149c99ffdd6d5df396462 (DEFAULT_TABLE table=default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.611585 3616 tablet_service.cc:1468] Processing CreateTablet for tablet b867925c426149c99ffdd6d5df396462 (DEFAULT_TABLE table=default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.611799 4040 log.cc:1199] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/wal/wals/0d8339cbcc774ea5bae71abeb3ecd2b7
I20250901 19:54:18.612077 3616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b867925c426149c99ffdd6d5df396462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.612083 3895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b867925c426149c99ffdd6d5df396462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.612259 4040 ts_tablet_manager.cc:1939] T 0d8339cbcc774ea5bae71abeb3ecd2b7 P e0064a1b18e5470b85bde2277757aac8: Deleting consensus metadata
I20250901 19:54:18.616437 3293 catalog_manager.cc:4928] TS e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271): tablet 0d8339cbcc774ea5bae71abeb3ecd2b7 (table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]) successfully deleted
I20250901 19:54:18.617421 3757 tablet_service.cc:1468] Processing CreateTablet for tablet b867925c426149c99ffdd6d5df396462 (DEFAULT_TABLE table=default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:18.617936 3757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b867925c426149c99ffdd6d5df396462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:18.619335 4036 tablet_bootstrap.cc:492] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:18.620923 3305 catalog_manager.cc:6652] ReplaceTablet: tablet 0d8339cbcc774ea5bae71abeb3ecd2b7 deleted and replaced by tablet b867925c426149c99ffdd6d5df396462
I20250901 19:54:18.627233 4037 tablet_bootstrap.cc:492] T b867925c426149c99ffdd6d5df396462 P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:18.629637 3305 catalog_manager.cc:5582] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193). New cstate: current_term: 1 leader_uuid: "c5b48d93671f4eb8980759b876e9ca21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20250901 19:54:18.630251 4036 tablet_bootstrap.cc:654] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.632987 4038 tablet_bootstrap.cc:492] T b867925c426149c99ffdd6d5df396462 P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:18.633580 4037 tablet_bootstrap.cc:654] T b867925c426149c99ffdd6d5df396462 P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.634850 4038 tablet_bootstrap.cc:654] T b867925c426149c99ffdd6d5df396462 P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:18.646759 3305 master_service.cc:948] ReplaceTablet: received request to replace tablet f640cc3c97394e16b88f7ef8148c19dd from {username='slave'} at 127.0.0.1:55634
I20250901 19:54:18.653931 3305 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55634:
table { table_name: "default.Table_3cbad8a0189443d9b60b0abc4460862a" } modify_external_catalogs: true
I20250901 19:54:18.655432 3305 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55634:
table { table_name: "default.Table_3cbad8a0189443d9b60b0abc4460862a" } modify_external_catalogs: true
I20250901 19:54:18.656525 4038 tablet_bootstrap.cc:492] T b867925c426149c99ffdd6d5df396462 P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:18.656821 4038 ts_tablet_manager.cc:1397] T b867925c426149c99ffdd6d5df396462 P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.007s
I20250901 19:54:18.657527 4036 tablet_bootstrap.cc:492] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:18.657752 4036 ts_tablet_manager.cc:1397] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.004s
I20250901 19:54:18.658010 4038 raft_consensus.cc:357] T b867925c426149c99ffdd6d5df396462 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.658330 4038 raft_consensus.cc:383] T b867925c426149c99ffdd6d5df396462 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.658458 4038 raft_consensus.cc:738] T b867925c426149c99ffdd6d5df396462 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.658635 4036 raft_consensus.cc:357] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.658846 4036 raft_consensus.cc:383] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.658933 4036 raft_consensus.cc:738] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.658887 4038 consensus_queue.cc:260] T b867925c426149c99ffdd6d5df396462 P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.659235 4036 consensus_queue.cc:260] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.660022 4036 ts_tablet_manager.cc:1428] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250901 19:54:18.662221 4038 ts_tablet_manager.cc:1428] T b867925c426149c99ffdd6d5df396462 P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250901 19:54:18.663399 4037 tablet_bootstrap.cc:492] T b867925c426149c99ffdd6d5df396462 P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:18.663903 4037 ts_tablet_manager.cc:1397] T b867925c426149c99ffdd6d5df396462 P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.008s
I20250901 19:54:18.664757 4037 raft_consensus.cc:357] T b867925c426149c99ffdd6d5df396462 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.665030 4037 raft_consensus.cc:383] T b867925c426149c99ffdd6d5df396462 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:18.665138 4037 raft_consensus.cc:738] T b867925c426149c99ffdd6d5df396462 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:18.665469 4037 consensus_queue.cc:260] T b867925c426149c99ffdd6d5df396462 P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.666078 4037 ts_tablet_manager.cc:1428] T b867925c426149c99ffdd6d5df396462 P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 19:54:18.702904 4065 raft_consensus.cc:491] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:18.703153 4065 raft_consensus.cc:513] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.704073 4065 leader_election.cc:290] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d397574faadb441192c99d530f14a734 (127.2.246.194:33557), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.705765 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1d516f8388c4a099f81d6e5a3c7cf4d" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734" is_pre_election: true
I20250901 19:54:18.706059 3778 raft_consensus.cc:2466] T e1d516f8388c4a099f81d6e5a3c7cf4d P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
I20250901 19:54:18.706550 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1d516f8388c4a099f81d6e5a3c7cf4d" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8" is_pre_election: true
I20250901 19:54:18.706763 3916 raft_consensus.cc:2466] T e1d516f8388c4a099f81d6e5a3c7cf4d P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
I20250901 19:54:18.707481 3572 leader_election.cc:304] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, e0064a1b18e5470b85bde2277757aac8; no voters:
I20250901 19:54:18.708463 4065 raft_consensus.cc:2802] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:18.708600 4065 raft_consensus.cc:491] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:18.708688 4065 raft_consensus.cc:3058] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.711584 4049 raft_consensus.cc:491] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:18.711812 4049 raft_consensus.cc:513] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.712700 4049 leader_election.cc:290] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d397574faadb441192c99d530f14a734 (127.2.246.194:33557), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.714111 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f37af30f3894824b7ce48101e1fda8a" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8" is_pre_election: true
I20250901 19:54:18.714366 3916 raft_consensus.cc:2466] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
I20250901 19:54:18.715019 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f37af30f3894824b7ce48101e1fda8a" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734" is_pre_election: true
I20250901 19:54:18.715272 3778 raft_consensus.cc:2466] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
I20250901 19:54:18.716133 3573 leader_election.cc:304] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters:
I20250901 19:54:18.717013 4049 raft_consensus.cc:2802] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:18.717126 4049 raft_consensus.cc:491] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:18.717226 4049 raft_consensus.cc:3058] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.720201 4049 raft_consensus.cc:513] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.721725 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f37af30f3894824b7ce48101e1fda8a" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734"
I20250901 19:54:18.721971 3778 raft_consensus.cc:3058] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.725003 3778 raft_consensus.cc:2466] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
I20250901 19:54:18.725744 3573 leader_election.cc:304] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters:
I20250901 19:54:18.726127 4066 raft_consensus.cc:2802] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:18.728094 4065 raft_consensus.cc:513] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.730301 3777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1d516f8388c4a099f81d6e5a3c7cf4d" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734"
I20250901 19:54:18.730597 3777 raft_consensus.cc:3058] T e1d516f8388c4a099f81d6e5a3c7cf4d P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.731436 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f37af30f3894824b7ce48101e1fda8a" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8"
I20250901 19:54:18.731663 3916 raft_consensus.cc:3058] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.733572 3777 raft_consensus.cc:2466] T e1d516f8388c4a099f81d6e5a3c7cf4d P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
I20250901 19:54:18.734442 3916 raft_consensus.cc:2466] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
I20250901 19:54:18.735477 3573 leader_election.cc:304] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters:
I20250901 19:54:18.741092 4049 leader_election.cc:290] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Requested vote from peers d397574faadb441192c99d530f14a734 (127.2.246.194:33557), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.742220 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1d516f8388c4a099f81d6e5a3c7cf4d" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8"
I20250901 19:54:18.742578 3916 raft_consensus.cc:3058] T e1d516f8388c4a099f81d6e5a3c7cf4d P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.746193 3916 raft_consensus.cc:2466] T e1d516f8388c4a099f81d6e5a3c7cf4d P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
I20250901 19:54:18.748548 4049 raft_consensus.cc:2802] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:18.748797 4049 raft_consensus.cc:695] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [term 1 LEADER]: Becoming Leader. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Running, Role: LEADER
I20250901 19:54:18.750535 4049 consensus_queue.cc:237] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.751852 4065 leader_election.cc:290] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Requested vote from peers d397574faadb441192c99d530f14a734 (127.2.246.194:33557), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.760082 4047 raft_consensus.cc:491] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:18.760322 4047 raft_consensus.cc:513] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.760999 4047 leader_election.cc:290] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.764138 3638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5067dd498ef4331bd9d9f3bcbcaec44" candidate_uuid: "d397574faadb441192c99d530f14a734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21" is_pre_election: true
I20250901 19:54:18.764395 3638 raft_consensus.cc:2466] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d397574faadb441192c99d530f14a734 in term 0.
I20250901 19:54:18.764866 3711 leader_election.cc:304] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters:
I20250901 19:54:18.765290 4047 raft_consensus.cc:2802] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:18.765410 4047 raft_consensus.cc:491] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:18.765527 4047 raft_consensus.cc:3058] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.768354 4047 raft_consensus.cc:513] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.769608 3638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5067dd498ef4331bd9d9f3bcbcaec44" candidate_uuid: "d397574faadb441192c99d530f14a734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21"
I20250901 19:54:18.769836 3638 raft_consensus.cc:3058] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.770462 4047 leader_election.cc:290] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 election: Requested vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.772706 3638 raft_consensus.cc:2466] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d397574faadb441192c99d530f14a734 in term 1.
I20250901 19:54:18.773324 3711 leader_election.cc:304] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters:
I20250901 19:54:18.773687 4047 raft_consensus.cc:2802] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:18.773844 4047 raft_consensus.cc:695] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [term 1 LEADER]: Becoming Leader. State: Replica: d397574faadb441192c99d530f14a734, State: Running, Role: LEADER
I20250901 19:54:18.774119 4047 consensus_queue.cc:237] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.761924 3915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5067dd498ef4331bd9d9f3bcbcaec44" candidate_uuid: "d397574faadb441192c99d530f14a734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8" is_pre_election: true
I20250901 19:54:18.779886 3915 raft_consensus.cc:2466] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d397574faadb441192c99d530f14a734 in term 0.
I20250901 19:54:18.770208 3914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5067dd498ef4331bd9d9f3bcbcaec44" candidate_uuid: "d397574faadb441192c99d530f14a734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8"
I20250901 19:54:18.780247 3914 raft_consensus.cc:3058] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.773061 4046 raft_consensus.cc:491] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:18.781193 4046 raft_consensus.cc:513] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.782114 4046 leader_election.cc:290] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:18.783766 3914 raft_consensus.cc:2466] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d397574faadb441192c99d530f14a734 in term 1.
I20250901 19:54:18.783813 3777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e43433b1ce9646e49962fe44bbed2491" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734" is_pre_election: true
I20250901 19:54:18.784024 3777 raft_consensus.cc:2466] T e43433b1ce9646e49962fe44bbed2491 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 0.
I20250901 19:54:18.784972 3851 leader_election.cc:304] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d397574faadb441192c99d530f14a734, e0064a1b18e5470b85bde2277757aac8; no voters:
I20250901 19:54:18.785267 4046 raft_consensus.cc:2802] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:18.785364 4046 raft_consensus.cc:491] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:18.785449 4046 raft_consensus.cc:3058] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.788136 4046 raft_consensus.cc:513] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.789268 3777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e43433b1ce9646e49962fe44bbed2491" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734"
I20250901 19:54:18.789479 3777 raft_consensus.cc:3058] T e43433b1ce9646e49962fe44bbed2491 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.792881 3637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e43433b1ce9646e49962fe44bbed2491" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21"
I20250901 19:54:18.793195 3637 raft_consensus.cc:3058] T e43433b1ce9646e49962fe44bbed2491 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.795746 4046 leader_election.cc:290] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 election: Requested vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:18.796162 3638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e43433b1ce9646e49962fe44bbed2491" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21" is_pre_election: true
I20250901 19:54:18.796294 3637 raft_consensus.cc:2466] T e43433b1ce9646e49962fe44bbed2491 P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 1.
I20250901 19:54:18.796972 3849 leader_election.cc:304] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, e0064a1b18e5470b85bde2277757aac8; no voters:
I20250901 19:54:18.798450 3777 raft_consensus.cc:2466] T e43433b1ce9646e49962fe44bbed2491 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 1.
I20250901 19:54:18.799252 4046 raft_consensus.cc:2802] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Leader election won for term 1
W20250901 19:54:18.803013 3573 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.246.194:33557, user_credentials={real_user=slave}} blocked reactor thread for 67553.1us
I20250901 19:54:18.808456 4066 raft_consensus.cc:695] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [term 1 LEADER]: Becoming Leader. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Running, Role: LEADER
I20250901 19:54:18.808925 4066 consensus_queue.cc:237] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.809695 4046 raft_consensus.cc:695] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [term 1 LEADER]: Becoming Leader. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Running, Role: LEADER
I20250901 19:54:18.810142 4046 consensus_queue.cc:237] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
19:54:18.843 [HMS - ERROR - pool-6-thread-2] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_c32254b687594ed89daf3edaded35200 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20250901 19:54:18.847011 3304 catalog_manager.cc:2145] Already present: failed to create HMS catalog entry for table [id=525b3a501d4a4b299debaaa43380a59d]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_c32254b687594ed89daf3edaded35200 already exists)
I20250901 19:54:18.847604 3304 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55634 (request call id 11) took 1417 ms. Trace:
I20250901 19:54:18.847767 3304 rpcz_store.cc:276] 0901 19:54:17.429811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 19:54:17.429860 (+ 49us) service_pool.cc:225] Handling call
0901 19:54:17.454631 (+ 24771us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 19:54:17.455046 (+ 415us) catalog_manager.cc:2122] Created new table and tablet info
0901 19:54:18.847552 (+1392506us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":978881,"Hive Metastore.run_cpu_time_us":1111,"Hive Metastore.run_wall_time_us":412680,"spinlock_wait_cycles":3584}
I20250901 19:54:18.861763 3304 catalog_manager.cc:5582] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0064a1b18e5470b85bde2277757aac8 (127.2.246.195). New cstate: current_term: 1 leader_uuid: "e0064a1b18e5470b85bde2277757aac8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } health_report { overall_health: HEALTHY } } }
I20250901 19:54:18.874819 3304 catalog_manager.cc:5582] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193). New cstate: current_term: 1 leader_uuid: "c5b48d93671f4eb8980759b876e9ca21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20250901 19:54:18.875654 3304 catalog_manager.cc:5582] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193). New cstate: current_term: 1 leader_uuid: "c5b48d93671f4eb8980759b876e9ca21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } health_report { overall_health: UNKNOWN } } }
F20250901 19:54:18.884114 3523 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_0a1ca85b493d4aa285d85b17df52c4aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_0a1ca85b493d4aa285d85b17df52c4aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0a1ca85b493d4aa285d85b17df52c4aa\",\"kudu.table_id\":\"fe4c5d5ccf384fbfbf05cb7b4f0a1bb6\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_c32254b687594ed89daf3edaded35200, message={"server":"","servicePrincipal":"","db":"default","table":"table_c32254b687594ed89daf3edaded35200","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c32254b687594ed89daf3edaded35200\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_c32254b687594ed89daf3edaded35200\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c32254b687594ed89daf3edaded35200\",\"kudu.table_id\":\"268741b95ddb403f8f59e4219723950e\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250901 19:54:18.887823 3304 catalog_manager.cc:5582] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 reported cstate change: term changed from 0 to 1, leader changed from <none> to d397574faadb441192c99d530f14a734 (127.2.246.194). New cstate: current_term: 1 leader_uuid: "d397574faadb441192c99d530f14a734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } health_report { overall_health: UNKNOWN } } }
@ 0x7f0032c62dcd google::LogMessage::Fail() at ??:0
@ 0x7f0032c66b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f0032c627cc google::LogMessage::Flush() at ??:0
@ 0x7f0032c62a11 google::LogMessage::~LogMessage() at ??:0
I20250901 19:54:18.899977 3304 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55634:
name: "default.Table_788e14a6980547248f5627f671a238a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f0049097794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f0049096041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f004909dcbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f004909db39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250901 19:54:18.918980 4066 raft_consensus.cc:491] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:18.919204 4066 raft_consensus.cc:513] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.920037 4066 leader_election.cc:290] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d397574faadb441192c99d530f14a734 (127.2.246.194:33557), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
@ 0x7f00359bfcc9 kudu::Thread::SuperviseThread() at ??:0
I20250901 19:54:18.921236 3914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b867925c426149c99ffdd6d5df396462" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8" is_pre_election: true
I20250901 19:54:18.921262 3777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b867925c426149c99ffdd6d5df396462" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734" is_pre_election: true
I20250901 19:54:18.921504 3914 raft_consensus.cc:2466] T b867925c426149c99ffdd6d5df396462 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
I20250901 19:54:18.921536 3777 raft_consensus.cc:2466] T b867925c426149c99ffdd6d5df396462 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
I20250901 19:54:18.922079 3572 leader_election.cc:304] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734, e0064a1b18e5470b85bde2277757aac8; no voters:
I20250901 19:54:18.922580 4066 raft_consensus.cc:2802] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:18.922667 4066 raft_consensus.cc:491] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:18.922763 4066 raft_consensus.cc:3058] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
@ 0x7f0038de16db start_thread at ??:0
I20250901 19:54:18.925899 4066 raft_consensus.cc:513] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.927419 3777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b867925c426149c99ffdd6d5df396462" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734"
I20250901 19:54:18.927732 3777 raft_consensus.cc:3058] T b867925c426149c99ffdd6d5df396462 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.931087 3777 raft_consensus.cc:2466] T b867925c426149c99ffdd6d5df396462 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
@ 0x7f0031b1371f clone at ??:0
I20250901 19:54:18.932376 3573 leader_election.cc:304] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters:
I20250901 19:54:18.932839 4100 raft_consensus.cc:2802] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:18.933079 4100 raft_consensus.cc:695] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [term 1 LEADER]: Becoming Leader. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Running, Role: LEADER
I20250901 19:54:18.933526 4100 consensus_queue.cc:237] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:18.934500 4066 leader_election.cc:290] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Requested vote from peers d397574faadb441192c99d530f14a734 (127.2.246.194:33557), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:18.936640 3914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b867925c426149c99ffdd6d5df396462" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8"
I20250901 19:54:18.936908 3914 raft_consensus.cc:3058] T b867925c426149c99ffdd6d5df396462 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:18.940279 3914 raft_consensus.cc:2466] T b867925c426149c99ffdd6d5df396462 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
W20250901 19:54:19.002665 4070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:19.003131 4070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:19.003362 4070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 19:54:19.013568 4070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 19:54:19.013752 4070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:19.013833 4070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 19:54:19.013899 4070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 19:54:19.029171 4070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.252:37003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.246.252
--webserver_port=36357
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:19.031384 4070 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:19.033726 4070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 19:54:19.047526 4123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:19.047704 4124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:19.055737 4126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:19.056543 4070 server_base.cc:1047] running on GCE node
I20250901 19:54:19.057545 4070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:19.064077 4046 consensus_queue.cc:1035] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:19.071906 4070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:19.073208 4070 hybrid_clock.cc:648] HybridClock initialized: now 1756756459073127 us; error 98 us; skew 500 ppm
I20250901 19:54:19.073678 4070 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:19.078208 4070 webserver.cc:480] Webserver started at http://127.2.246.252:36357/ using document root <none> and password file <none>
I20250901 19:54:19.078832 4103 consensus_queue.cc:1035] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:19.079989 4070 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:19.080161 4070 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:19.088444 4070 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20250901 19:54:19.092707 4138 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:19.094728 4070 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250901 19:54:19.094942 4070 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
uuid: "08076b584d2a4aec8fd9868f85c9c7cd"
format_stamp: "Formatted at 2025-09-01 19:54:14 on dist-test-slave-jnn4"
I20250901 19:54:19.096032 4070 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:19.125828 4070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:19.126767 4070 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:19.127060 4070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:19.152215 4070 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.252:37003
I20250901 19:54:19.152256 4189 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.252:37003 every 8 connection(s)
I20250901 19:54:19.154213 4070 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
I20250901 19:54:19.160297 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 4070
I20250901 19:54:19.164744 4190 sys_catalog.cc:263] Verifying existing consensus state
I20250901 19:54:19.171324 4190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Bootstrap starting.
I20250901 19:54:19.183977 4190 log.cc:826] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:19.196177 4190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 19:54:19.196888 4190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Bootstrap complete.
W20250901 19:54:19.197598 3713 connection.cc:537] client connection to 127.2.246.254:33523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:19.197654 3851 connection.cc:537] client connection to 127.2.246.254:33523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:19.198447 3973 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.246.254:33523)
W20250901 19:54:19.199395 3573 connection.cc:537] client connection to 127.2.246.254:33523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:19.200598 4022 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 19:54:19.198489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.254:33523 (local address 127.2.246.194:53315)
0901 19:54:19.199568 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0901 19:54:19.199693 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":943}
W20250901 19:54:19.202386 3973 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.246.254:33523)
W20250901 19:54:19.203334 3973 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.246.254:33523)
W20250901 19:54:19.204555 3973 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.246.254:33523)
I20250901 19:54:19.206912 4190 raft_consensus.cc:357] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:19.208298 4190 raft_consensus.cc:738] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08076b584d2a4aec8fd9868f85c9c7cd, State: Initialized, Role: FOLLOWER
I20250901 19:54:19.209360 4190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:19.212980 4196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:19.213335 4196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:19.214774 4190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: configured and running, proceeding with master startup.
I20250901 19:54:19.242749 4210 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:19.242874 4210 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: loading cluster ID for follower catalog manager: success
I20250901 19:54:19.239833 4070 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:54:19.244103 4070 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:54:19.247962 4210 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: acquiring CA information for follower catalog manager: success
I20250901 19:54:19.250933 4210 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 19:54:19.308298 4100 consensus_queue.cc:1035] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:19.319819 3961 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
I20250901 19:54:19.320079 3961 heartbeater.cc:499] Master 127.2.246.252:37003 was elected leader, sending a full tablet report...
I20250901 19:54:19.326023 4155 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" instance_seqno: 1756756457020957) as {username='slave'} at 127.2.246.195:51675; Asking this server to re-register.
I20250901 19:54:19.327400 3961 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:19.330192 4154 ts_manager.cc:194] Registered new tserver with Master: e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:19.333154 4100 consensus_queue.cc:1035] T f4fe9cadf6414b548cbd95572d9ed1c8 P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:19.337430 3683 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
I20250901 19:54:19.340237 4154 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" instance_seqno: 1756756455802456) as {username='slave'} at 127.2.246.193:57469; Asking this server to re-register.
I20250901 19:54:19.341164 3683 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:19.343878 4155 ts_manager.cc:194] Registered new tserver with Master: c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:19.348965 3823 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
I20250901 19:54:19.351459 4155 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d397574faadb441192c99d530f14a734" instance_seqno: 1756756456460392) as {username='slave'} at 127.2.246.194:42841; Asking this server to re-register.
I20250901 19:54:19.352367 3823 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:19.354105 4155 ts_manager.cc:194] Registered new tserver with Master: d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:19.386125 4047 consensus_queue.cc:1035] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:19.395812 4047 consensus_queue.cc:1035] T 516ed4484b174a8d8cb8c6b9f076d31e P d397574faadb441192c99d530f14a734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:19.405428 4222 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:54:19.405661 4222 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:19.406389 4222 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:19.407567 3364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250901 19:54:19.408891 3364 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
I20250901 19:54:19.414630 4165 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" candidate_term: 3 candidate_status { last_received { term: 2 index: 38 } } ignore_live_leader: false dest_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" is_pre_election: true
I20250901 19:54:19.415237 4165 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d944ed54f8c43c9b8c4e7d589dad395 in term 1.
I20250901 19:54:19.415951 3363 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 4c9163569ae1418090a245256e263186
I20250901 19:54:19.416385 4222 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250901 19:54:19.416512 4222 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:54:19.416618 4222 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 2 FOLLOWER]: Advancing to term 3
I20250901 19:54:19.418668 4222 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:19.419456 4222 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 3 election: Requested vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
I20250901 19:54:19.420248 4165 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" candidate_term: 3 candidate_status { last_received { term: 2 index: 38 } } ignore_live_leader: false dest_uuid: "08076b584d2a4aec8fd9868f85c9c7cd"
I20250901 19:54:19.420496 4165 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 1 FOLLOWER]: Advancing to term 3
W20250901 19:54:19.421069 3364 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
I20250901 19:54:19.426498 4165 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d944ed54f8c43c9b8c4e7d589dad395 in term 3.
I20250901 19:54:19.427109 3363 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 4c9163569ae1418090a245256e263186
I20250901 19:54:19.427554 4222 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 3 FOLLOWER]: Leader election won for term 3
I20250901 19:54:19.428073 4222 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 3 LEADER]: Becoming Leader. State: Replica: 1d944ed54f8c43c9b8c4e7d589dad395, State: Running, Role: LEADER
I20250901 19:54:19.428539 4222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 2.38, Last appended by leader: 38, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:19.429976 4226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d944ed54f8c43c9b8c4e7d589dad395. Latest consensus state: current_term: 3 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:19.430285 4226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:19.430992 4227 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 19:54:19.433753 4227 catalog_manager.cc:671] Loaded metadata for table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077]
I20250901 19:54:19.434305 4227 catalog_manager.cc:671] Loaded metadata for table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]
I20250901 19:54:19.434674 4227 catalog_manager.cc:671] Loaded metadata for table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]
I20250901 19:54:19.435106 4227 catalog_manager.cc:671] Loaded metadata for table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67]
I20250901 19:54:19.435472 4227 catalog_manager.cc:671] Loaded metadata for table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]
I20250901 19:54:19.435917 4227 catalog_manager.cc:671] Loaded metadata for table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]
I20250901 19:54:19.436318 4227 catalog_manager.cc:671] Loaded metadata for table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8]
I20250901 19:54:19.436662 4227 catalog_manager.cc:671] Loaded metadata for table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616]
I20250901 19:54:19.436985 4227 catalog_manager.cc:671] Loaded metadata for table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]
I20250901 19:54:19.439174 4227 tablet_loader.cc:96] loaded metadata for tablet 1d21a96219ab4f5f96bc6d9976f990ff (table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896])
I20250901 19:54:19.439503 4227 tablet_loader.cc:96] loaded metadata for tablet 516ed4484b174a8d8cb8c6b9f076d31e (table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8])
I20250901 19:54:19.439951 4227 tablet_loader.cc:96] loaded metadata for tablet 5f37af30f3894824b7ce48101e1fda8a (table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59])
I20250901 19:54:19.440286 4227 tablet_loader.cc:96] loaded metadata for tablet 7ab5ab06eece48fdb346f232247f72e0 (table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077])
I20250901 19:54:19.440604 4227 tablet_loader.cc:96] loaded metadata for tablet a5067dd498ef4331bd9d9f3bcbcaec44 (table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612])
I20250901 19:54:19.440907 4227 tablet_loader.cc:96] loaded metadata for tablet b867925c426149c99ffdd6d5df396462 (table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6])
I20250901 19:54:19.441229 4227 tablet_loader.cc:96] loaded metadata for tablet e1d516f8388c4a099f81d6e5a3c7cf4d (table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3])
I20250901 19:54:19.441510 4227 tablet_loader.cc:96] loaded metadata for tablet e43433b1ce9646e49962fe44bbed2491 (table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67])
I20250901 19:54:19.441792 4227 tablet_loader.cc:96] loaded metadata for tablet f4fe9cadf6414b548cbd95572d9ed1c8 (table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616])
I20250901 19:54:19.442273 4227 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 19:54:19.443384 4227 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:19.443461 4227 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 19:54:19.444789 4227 catalog_manager.cc:1506] Loading token signing keys...
I20250901 19:54:19.445870 4227 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Loaded TSK: 0
I20250901 19:54:19.446887 4227 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 19:54:19.447851 4227 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 19:54:19.448932 4227 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250901 19:54:19.453902 3378 master_service.cc:948] ReplaceTablet: received request to replace tablet 516ed4484b174a8d8cb8c6b9f076d31e from {username='slave'} at 127.0.0.1:58976
I20250901 19:54:19.454054 3376 master_service.cc:948] ReplaceTablet: received request to replace tablet e43433b1ce9646e49962fe44bbed2491 from {username='slave'} at 127.0.0.1:58976
I20250901 19:54:19.454430 3374 master_service.cc:948] ReplaceTablet: received request to replace tablet 5f37af30f3894824b7ce48101e1fda8a from {username='slave'} at 127.0.0.1:58976
I20250901 19:54:19.456454 3375 master_service.cc:948] ReplaceTablet: received request to replace tablet e43433b1ce9646e49962fe44bbed2491 from {username='slave'} at 127.0.0.1:58976
I20250901 19:54:19.455865 3373 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58976:
name: "default.Table_4c54782dd2bb4c66a3be449970979d0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:19.457746 3369 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58976:
name: "default.Table_69070232fcfc4853b5fff18e23fbf965"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:19.457381 3371 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58976:
table { table_name: "default.Table_51c52d3eec714bbd9a77bed453c02bb1" } modify_external_catalogs: true
I20250901 19:54:19.461478 3371 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58976:
table { table_name: "default.Table_51c52d3eec714bbd9a77bed453c02bb1" } modify_external_catalogs: true
I20250901 19:54:19.454972 3377 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58976:
name: "default.Table_14959779c0504cb4bf46faa666221361"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:19.456233 3372 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58976:
name: "default.Table_240cfd071a7c475bb44ff628d903bf9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:19.457203 3370 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58976:
name: "default.Table_8728509cfe374d5ea49110ea201c1659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250901 19:54:19.468663 3552 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_0a1ca85b493d4aa285d85b17df52c4aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_0a1ca85b493d4aa285d85b17df52c4aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0a1ca85b493d4aa285d85b17df52c4aa\",\"kudu.table_id\":\"fe4c5d5ccf384fbfbf05cb7b4f0a1bb6\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_c32254b687594ed89daf3edaded35200, message={"server":"","servicePrincipal":"","db":"default","table":"table_c32254b687594ed89daf3edaded35200","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c32254b687594ed89daf3edaded35200\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_c32254b687594ed89daf3edaded35200\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c32254b687594ed89daf3edaded35200\",\"kudu.table_id\":\"268741b95ddb403f8f59e4219723950e\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f64bebd9dcd google::LogMessage::Fail() at ??:0
@ 0x7f64bebddb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f64bebd97cc google::LogMessage::Flush() at ??:0
@ 0x7f64bebd9a11 google::LogMessage::~LogMessage() at ??:0
I20250901 19:54:19.476577 4165 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Refusing update from remote peer 1d944ed54f8c43c9b8c4e7d589dad395: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 40. (index mismatch)
I20250901 19:54:19.477802 4226 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 4, Time since last communication: 0.000s
@ 0x7f64d500e794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250901 19:54:19.484813 4100 consensus_queue.cc:1035] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 19:54:19.492167 3364 consensus_peers.cc:489] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 -> Peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Couldn't send request to peer 4c9163569ae1418090a245256e263186. Status: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f64d500d041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f64d5014cbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250901 19:54:19.513833 4066 consensus_queue.cc:1035] T 7ab5ab06eece48fdb346f232247f72e0 P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0x7f64d5014b39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f64c1936cc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f64c4d586db start_thread at ??:0
@ 0x7f64bda8a71f clone at ??:0
I20250901 19:54:19.540113 4196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d944ed54f8c43c9b8c4e7d589dad395. Latest consensus state: current_term: 3 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:19.540405 4196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:19.541111 4240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:19.541429 4240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:19.584517 4231 mvcc.cc:204] Tried to move back new op lower bound from 7195674455613026304 to 7195674449407266816. Current Snapshot: MvccSnapshot[applied={T|T < 7195674455613026304 or (T in {7195674455613026304})}]
I20250901 19:54:19.624260 4103 consensus_queue.cc:1035] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:19.632207 4250 consensus_queue.cc:1035] T e43433b1ce9646e49962fe44bbed2491 P e0064a1b18e5470b85bde2277757aac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 19:54:19.752334 4140 connection.cc:537] server connection from 127.0.0.1:35184 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 19:54:19.863269 4066 consensus_queue.cc:1035] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:19.869278 4099 consensus_queue.cc:1035] T e1d516f8388c4a099f81d6e5a3c7cf4d P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:19.936021 4220 consensus_queue.cc:1035] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:19.942761 4220 consensus_queue.cc:1035] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:19.949512 4099 consensus_queue.cc:1035] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:19.956307 4099 consensus_queue.cc:1035] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:19.985155 4240 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:19.985394 4240 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:19.987120 4240 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:19.988310 4141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:19.990041 4141 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:19.990249 4141 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:19.990356 4141 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:19.990665 4240 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250901 19:54:20.108701 4099 consensus_queue.cc:1035] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:20.113708 4066 consensus_queue.cc:1035] T b867925c426149c99ffdd6d5df396462 P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:20.562599 4275 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:20.562815 4275 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:20.563481 4275 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:20.573067 4141 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:20.573290 4141 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:20.573410 4141 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:20.573715 4275 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250901 19:54:21.494926 4289 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:21.495165 4289 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:21.495960 4289 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:21.498410 4141 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:21.498677 4141 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:21.498764 4141 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:21.499082 4289 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250901 19:54:22.342790 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.254:33523
--webserver_interface=127.2.246.254
--webserver_port=40931
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250901 19:54:22.622048 4304 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:22.622244 4304 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:22.623258 4304 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:22.629832 4141 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:22.630049 4141 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:22.630177 4141 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:22.630451 4304 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250901 19:54:22.701457 4303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:22.701850 4303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:22.701984 4303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 19:54:22.710566 4303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 19:54:22.710701 4303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:22.710791 4303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 19:54:22.710860 4303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 19:54:22.721475 4303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.254:33523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.254
--webserver_port=40931
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:22.723361 4303 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:22.725697 4303 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 19:54:22.734289 4311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:22.734381 4312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:22.734992 4314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:22.735834 4303 server_base.cc:1047] running on GCE node
I20250901 19:54:22.736667 4303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:22.738253 4303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:22.739477 4303 hybrid_clock.cc:648] HybridClock initialized: now 1756756462739441 us; error 41 us; skew 500 ppm
I20250901 19:54:22.739897 4303 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:22.742344 4303 webserver.cc:480] Webserver started at http://127.2.246.254:40931/ using document root <none> and password file <none>
I20250901 19:54:22.743140 4303 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:22.743274 4303 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:22.750412 4303 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20250901 19:54:22.755336 4320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:22.757433 4303 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250901 19:54:22.757637 4303 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
uuid: "4c9163569ae1418090a245256e263186"
format_stamp: "Formatted at 2025-09-01 19:54:13 on dist-test-slave-jnn4"
I20250901 19:54:22.758486 4303 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:22.778376 4303 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:22.779402 4303 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:22.779758 4303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:22.805784 4303 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.254:33523
I20250901 19:54:22.805825 4374 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.254:33523 every 8 connection(s)
I20250901 19:54:22.807824 4303 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data/info.pb
I20250901 19:54:22.814925 4375 sys_catalog.cc:263] Verifying existing consensus state
I20250901 19:54:22.815601 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 4303
I20250901 19:54:22.822542 4375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Bootstrap starting.
I20250901 19:54:22.846006 4375 log.cc:826] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:22.856127 3824 heartbeater.cc:344] Connected to a master server at 127.2.246.254:33523
I20250901 19:54:22.883754 4375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=24 ignored=0} mutations{seen=25 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 19:54:22.884392 4375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Bootstrap complete.
I20250901 19:54:22.889720 4375 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:22.890537 4375 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9163569ae1418090a245256e263186, State: Initialized, Role: FOLLOWER
I20250901 19:54:22.891280 4375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 2.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:22.892131 4382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:22.892381 4382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:22.893236 4375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 19:54:22.907056 4396 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:22.907232 4396 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: loading cluster ID for follower catalog manager: success
I20250901 19:54:22.911968 4396 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: acquiring CA information for follower catalog manager: success
I20250901 19:54:22.915336 4396 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 19:54:22.916261 4303 master_runner.cc:186] Error getting master registration for 127.2.246.253:34127: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111),
I20250901 19:54:22.922158 4303 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:54:23.280498 3684 heartbeater.cc:344] Connected to a master server at 127.2.246.254:33523
I20250901 19:54:23.283460 4338 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" instance_seqno: 1756756455802456) as {username='slave'} at 127.2.246.193:37633; Asking this server to re-register.
I20250901 19:54:23.284302 3684 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:23.286275 4338 ts_manager.cc:194] Registered new tserver with Master: c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:23.305037 3962 heartbeater.cc:344] Connected to a master server at 127.2.246.254:33523
I20250901 19:54:23.306968 4338 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" instance_seqno: 1756756457020957) as {username='slave'} at 127.2.246.195:41225; Asking this server to re-register.
I20250901 19:54:23.307811 3962 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:23.309083 4338 ts_manager.cc:194] Registered new tserver with Master: e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:23.319242 4382 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:23.319510 4382 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:23.321501 4382 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:23.322170 4323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:54:23.324119 4323 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:23.328612 4165 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c9163569ae1418090a245256e263186" candidate_term: 3 candidate_status { last_received { term: 2 index: 38 } } ignore_live_leader: false dest_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" is_pre_election: true
I20250901 19:54:23.328950 4165 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c9163569ae1418090a245256e263186 in current term 3: Already voted for candidate 1d944ed54f8c43c9b8c4e7d589dad395 in this term.
I20250901 19:54:23.329614 4322 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c9163569ae1418090a245256e263186; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395
I20250901 19:54:23.330001 4382 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 2 FOLLOWER]: Advancing to term 3
I20250901 19:54:23.333930 4382 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250901 19:54:23.362134 4338 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d397574faadb441192c99d530f14a734" instance_seqno: 1756756456460392) as {username='slave'} at 127.2.246.194:51625; Asking this server to re-register.
I20250901 19:54:23.362936 3824 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:23.364480 4338 ts_manager.cc:194] Registered new tserver with Master: d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:23.512264 4406 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:23.512468 4406 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:23.513199 4406 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:23.516017 4141 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:23.519974 4349 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "4c9163569ae1418090a245256e263186" is_pre_election: true
I20250901 19:54:23.520354 4349 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08076b584d2a4aec8fd9868f85c9c7cd in term 3.
I20250901 19:54:23.520982 4141 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186; no voters: 1d944ed54f8c43c9b8c4e7d589dad395
I20250901 19:54:23.521311 4406 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250901 19:54:23.521409 4406 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:23.521507 4406 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 3 FOLLOWER]: Advancing to term 4
I20250901 19:54:23.523438 4406 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:23.524035 4406 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 election: Requested vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
I20250901 19:54:23.524513 4349 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "4c9163569ae1418090a245256e263186"
I20250901 19:54:23.524679 4349 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 3 FOLLOWER]: Advancing to term 4
W20250901 19:54:23.525758 4141 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:23.527539 4349 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08076b584d2a4aec8fd9868f85c9c7cd in term 4.
I20250901 19:54:23.528017 4141 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186; no voters: 1d944ed54f8c43c9b8c4e7d589dad395
I20250901 19:54:23.528304 4406 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 4 FOLLOWER]: Leader election won for term 4
I20250901 19:54:23.528739 4406 raft_consensus.cc:695] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 4 LEADER]: Becoming Leader. State: Replica: 08076b584d2a4aec8fd9868f85c9c7cd, State: Running, Role: LEADER
I20250901 19:54:23.529224 4406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 3.40, Last appended by leader: 40, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:23.530900 4410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08076b584d2a4aec8fd9868f85c9c7cd. Latest consensus state: current_term: 4 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:23.531216 4410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: LEADER
W20250901 19:54:24.070650 4141 consensus_peers.cc:489] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd -> Peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Couldn't send request to peer 1d944ed54f8c43c9b8c4e7d589dad395. Status: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 19:54:24.109450 4349 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 4 FOLLOWER]: Refusing update from remote peer 08076b584d2a4aec8fd9868f85c9c7cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 4 index: 41. (index mismatch)
I20250901 19:54:24.110360 4414 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 38, Time since last communication: 0.000s
I20250901 19:54:24.115682 4417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08076b584d2a4aec8fd9868f85c9c7cd. Latest consensus state: current_term: 4 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:24.116088 4417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:24.120019 4414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:24.120297 4414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:24.120434 4420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:24.120662 4420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:24.121479 4417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:24.121452 4425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:24.121735 4417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:24.121795 4425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:24.177026 4412 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 19:54:24.179831 4412 catalog_manager.cc:671] Loaded metadata for table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077]
I20250901 19:54:24.180294 4412 catalog_manager.cc:671] Loaded metadata for table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]
I20250901 19:54:24.180605 4412 catalog_manager.cc:671] Loaded metadata for table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]
I20250901 19:54:24.180918 4412 catalog_manager.cc:671] Loaded metadata for table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67]
I20250901 19:54:24.181216 4412 catalog_manager.cc:671] Loaded metadata for table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]
I20250901 19:54:24.181514 4412 catalog_manager.cc:671] Loaded metadata for table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]
I20250901 19:54:24.181805 4412 catalog_manager.cc:671] Loaded metadata for table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8]
I20250901 19:54:24.182106 4412 catalog_manager.cc:671] Loaded metadata for table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616]
I20250901 19:54:24.182377 4412 catalog_manager.cc:671] Loaded metadata for table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]
I20250901 19:54:24.184542 4412 tablet_loader.cc:96] loaded metadata for tablet 1d21a96219ab4f5f96bc6d9976f990ff (table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896])
I20250901 19:54:24.184995 4412 tablet_loader.cc:96] loaded metadata for tablet 5f37af30f3894824b7ce48101e1fda8a (table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59])
I20250901 19:54:24.185246 4412 tablet_loader.cc:96] loaded metadata for tablet 7ab5ab06eece48fdb346f232247f72e0 (table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077])
I20250901 19:54:24.185484 4412 tablet_loader.cc:96] loaded metadata for tablet a5067dd498ef4331bd9d9f3bcbcaec44 (table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612])
I20250901 19:54:24.185793 4412 tablet_loader.cc:96] loaded metadata for tablet b867925c426149c99ffdd6d5df396462 (table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6])
I20250901 19:54:24.186009 4412 tablet_loader.cc:96] loaded metadata for tablet e003ca1913284f239f5b0474fc621abe (table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8])
I20250901 19:54:24.186249 4412 tablet_loader.cc:96] loaded metadata for tablet e1d516f8388c4a099f81d6e5a3c7cf4d (table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3])
I20250901 19:54:24.186480 4412 tablet_loader.cc:96] loaded metadata for tablet e43433b1ce9646e49962fe44bbed2491 (table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67])
I20250901 19:54:24.186784 4412 tablet_loader.cc:96] loaded metadata for tablet f4fe9cadf6414b548cbd95572d9ed1c8 (table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616])
I20250901 19:54:24.187191 4412 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 19:54:24.188218 4412 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:24.188298 4412 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 19:54:24.189507 4412 catalog_manager.cc:1506] Loading token signing keys...
I20250901 19:54:24.190378 4412 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Loaded TSK: 0
I20250901 19:54:24.191334 4412 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 19:54:24.192261 4412 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 19:54:24.193082 4412 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
F20250901 19:54:24.232299 4209 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_0a1ca85b493d4aa285d85b17df52c4aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_0a1ca85b493d4aa285d85b17df52c4aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0a1ca85b493d4aa285d85b17df52c4aa\",\"kudu.table_id\":\"fe4c5d5ccf384fbfbf05cb7b4f0a1bb6\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_c32254b687594ed89daf3edaded35200, message={"server":"","servicePrincipal":"","db":"default","table":"table_c32254b687594ed89daf3edaded35200","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c32254b687594ed89daf3edaded35200\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_c32254b687594ed89daf3edaded35200\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c32254b687594ed89daf3edaded35200\",\"kudu.table_id\":\"268741b95ddb403f8f59e4219723950e\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fdce6771dcd google::LogMessage::Fail() at ??:0
@ 0x7fdce6775b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fdce67717cc google::LogMessage::Flush() at ??:0
@ 0x7fdce6771a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fdcfcba6794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fdcfcba5041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250901 19:54:24.250238 4070 master_runner.cc:186] Error getting master registration for 127.2.246.253:34127: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111),
I20250901 19:54:24.250471 4070 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
@ 0x7fdcfcbaccbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fdcfcbacb39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7fdce94cecc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdcec8f06db start_thread at ??:0
@ 0x7fdce562271f clone at ??:0
W20250901 19:54:24.439363 4321 connection.cc:537] server connection from 127.0.0.1:32916 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:24.439357 3712 connection.cc:537] client connection to 127.2.246.252:37003 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:24.439862 3972 connection.cc:537] client connection to 127.2.246.252:37003 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:24.440073 3572 connection.cc:537] client connection to 127.2.246.252:37003 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:24.440245 3850 connection.cc:537] client connection to 127.2.246.252:37003 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:24.440661 3972 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.246.252:37003)
W20250901 19:54:24.440943 3972 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.246.252:37003)
W20250901 19:54:24.441361 3972 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.246.252:37003)
W20250901 19:54:24.441859 3972 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.246.252:37003)
I20250901 19:54:24.767587 4441 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:24.767848 4441 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:24.768586 4441 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:24.770720 4322 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
W20250901 19:54:24.770927 4323 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:24.771041 4323 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c9163569ae1418090a245256e263186; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395
I20250901 19:54:24.771262 4441 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250901 19:54:25.436225 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.253:34127
--webserver_interface=127.2.246.253
--webserver_port=36885
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250901 19:54:25.518513 4449 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:25.518755 4449 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:25.519560 4449 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:25.521842 4322 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
W20250901 19:54:25.522045 4323 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:25.522182 4323 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c9163569ae1418090a245256e263186; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395
I20250901 19:54:25.522543 4449 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250901 19:54:25.792721 4448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:25.793107 4448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:25.793227 4448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 19:54:25.802084 4448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 19:54:25.802222 4448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:25.802299 4448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 19:54:25.802364 4448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 19:54:25.813803 4448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.253:34127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.246.253
--webserver_port=36885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:25.816010 4448 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:25.818317 4448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 19:54:25.827760 4458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:25.827965 4457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:25.829460 4460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:25.829866 4448 server_base.cc:1047] running on GCE node
I20250901 19:54:25.830665 4448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:25.832245 4448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:25.833460 4448 hybrid_clock.cc:648] HybridClock initialized: now 1756756465833408 us; error 41 us; skew 500 ppm
I20250901 19:54:25.833814 4448 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:25.836223 4448 webserver.cc:480] Webserver started at http://127.2.246.253:36885/ using document root <none> and password file <none>
I20250901 19:54:25.837002 4448 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:25.837132 4448 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:25.842931 4448 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250901 19:54:25.846419 4466 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:25.848140 4448 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250901 19:54:25.848438 4448 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
uuid: "1d944ed54f8c43c9b8c4e7d589dad395"
format_stamp: "Formatted at 2025-09-01 19:54:14 on dist-test-slave-jnn4"
I20250901 19:54:25.849469 4448 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:25.878938 4448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:25.879853 4448 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:25.880107 4448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:25.900703 4448 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.253:34127
I20250901 19:54:25.900771 4517 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.253:34127 every 8 connection(s)
I20250901 19:54:25.902526 4448 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/info.pb
I20250901 19:54:25.908142 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 4448
I20250901 19:54:25.910486 4518 sys_catalog.cc:263] Verifying existing consensus state
I20250901 19:54:25.914479 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Bootstrap starting.
I20250901 19:54:25.928664 4518 log.cc:826] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:25.964159 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=38 ignored=0} inserts{seen=24 ignored=0} mutations{seen=25 ignored=0} orphaned_commits=0. Pending: 4 replicates
I20250901 19:54:25.964843 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Bootstrap complete.
I20250901 19:54:25.970268 4518 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 3 FOLLOWER]: Replica starting. Triggering 4 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:25.973516 4518 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d944ed54f8c43c9b8c4e7d589dad395, State: Initialized, Role: FOLLOWER
I20250901 19:54:25.974325 4518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 3.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:25.975320 4523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:25.975739 4523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:25.976547 4518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 19:54:26.011430 4537 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:26.011683 4537 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: loading cluster ID for follower catalog manager: success
I20250901 19:54:26.016413 4537 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: acquiring CA information for follower catalog manager: success
I20250901 19:54:26.019515 4537 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 19:54:26.030099 4448 master_runner.cc:186] Error getting master registration for 127.2.246.252:37003: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111),
I20250901 19:54:26.035328 4448 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:54:26.289479 3685 heartbeater.cc:344] Connected to a master server at 127.2.246.253:34127
I20250901 19:54:26.293108 4482 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" instance_seqno: 1756756455802456) as {username='slave'} at 127.2.246.193:59949; Asking this server to re-register.
I20250901 19:54:26.293967 3685 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:26.295882 4482 ts_manager.cc:194] Registered new tserver with Master: c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:26.311410 3963 heartbeater.cc:344] Connected to a master server at 127.2.246.253:34127
I20250901 19:54:26.313462 4482 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" instance_seqno: 1756756457020957) as {username='slave'} at 127.2.246.195:40795; Asking this server to re-register.
I20250901 19:54:26.314256 3963 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:26.315809 4482 ts_manager.cc:194] Registered new tserver with Master: e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:26.324231 4543 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:26.324466 4543 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:26.325264 4543 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:26.328006 4322 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:26.333462 4493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c9163569ae1418090a245256e263186" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" is_pre_election: true
I20250901 19:54:26.334060 4493 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c9163569ae1418090a245256e263186 in term 3.
I20250901 19:54:26.334674 4323 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186; no voters: 08076b584d2a4aec8fd9868f85c9c7cd
I20250901 19:54:26.335038 4543 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250901 19:54:26.335176 4543 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:26.335270 4543 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 4 FOLLOWER]: Advancing to term 5
I20250901 19:54:26.337378 4543 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:26.338001 4543 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 5 election: Requested vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
I20250901 19:54:26.338640 4493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c9163569ae1418090a245256e263186" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "1d944ed54f8c43c9b8c4e7d589dad395"
I20250901 19:54:26.338855 4493 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 3 FOLLOWER]: Advancing to term 5
W20250901 19:54:26.339921 4322 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:26.343181 4493 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c9163569ae1418090a245256e263186 in term 5.
I20250901 19:54:26.343855 4323 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186; no voters: 08076b584d2a4aec8fd9868f85c9c7cd
I20250901 19:54:26.344264 4543 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 5 FOLLOWER]: Leader election won for term 5
I20250901 19:54:26.344736 4543 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 5 LEADER]: Becoming Leader. State: Replica: 4c9163569ae1418090a245256e263186, State: Running, Role: LEADER
I20250901 19:54:26.345263 4543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:26.348217 4547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c9163569ae1418090a245256e263186. Latest consensus state: current_term: 5 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:26.348731 4547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:26.349648 4549 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 19:54:26.353004 4549 catalog_manager.cc:671] Loaded metadata for table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077]
I20250901 19:54:26.353560 4549 catalog_manager.cc:671] Loaded metadata for table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]
I20250901 19:54:26.354032 4549 catalog_manager.cc:671] Loaded metadata for table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]
I20250901 19:54:26.354491 4549 catalog_manager.cc:671] Loaded metadata for table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67]
I20250901 19:54:26.354920 4549 catalog_manager.cc:671] Loaded metadata for table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]
I20250901 19:54:26.355379 4549 catalog_manager.cc:671] Loaded metadata for table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]
I20250901 19:54:26.355904 4549 catalog_manager.cc:671] Loaded metadata for table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8]
I20250901 19:54:26.356307 4549 catalog_manager.cc:671] Loaded metadata for table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616]
I20250901 19:54:26.356616 4549 catalog_manager.cc:671] Loaded metadata for table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]
I20250901 19:54:26.358596 3829 heartbeater.cc:344] Connected to a master server at 127.2.246.253:34127
I20250901 19:54:26.359455 4549 tablet_loader.cc:96] loaded metadata for tablet 1d21a96219ab4f5f96bc6d9976f990ff (table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896])
I20250901 19:54:26.360143 4549 tablet_loader.cc:96] loaded metadata for tablet 5f37af30f3894824b7ce48101e1fda8a (table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59])
I20250901 19:54:26.360550 4549 tablet_loader.cc:96] loaded metadata for tablet 7ab5ab06eece48fdb346f232247f72e0 (table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077])
I20250901 19:54:26.360880 4549 tablet_loader.cc:96] loaded metadata for tablet a5067dd498ef4331bd9d9f3bcbcaec44 (table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612])
I20250901 19:54:26.361248 4549 tablet_loader.cc:96] loaded metadata for tablet b867925c426149c99ffdd6d5df396462 (table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6])
I20250901 19:54:26.361268 4482 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d397574faadb441192c99d530f14a734" instance_seqno: 1756756456460392) as {username='slave'} at 127.2.246.194:45437; Asking this server to re-register.
I20250901 19:54:26.361621 4549 tablet_loader.cc:96] loaded metadata for tablet e003ca1913284f239f5b0474fc621abe (table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8])
I20250901 19:54:26.362003 4549 tablet_loader.cc:96] loaded metadata for tablet e1d516f8388c4a099f81d6e5a3c7cf4d (table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3])
I20250901 19:54:26.362167 3829 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:26.362363 4549 tablet_loader.cc:96] loaded metadata for tablet e43433b1ce9646e49962fe44bbed2491 (table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67])
I20250901 19:54:26.362766 4549 tablet_loader.cc:96] loaded metadata for tablet f4fe9cadf6414b548cbd95572d9ed1c8 (table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616])
I20250901 19:54:26.363359 4549 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 19:54:26.364446 4482 ts_manager.cc:194] Registered new tserver with Master: d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:26.364768 4549 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:26.364853 4549 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 19:54:26.366449 4549 catalog_manager.cc:1506] Loading token signing keys...
I20250901 19:54:26.367780 4549 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Loaded TSK: 0
I20250901 19:54:26.368819 4549 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 19:54:26.369808 4549 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 19:54:26.370683 4549 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250901 19:54:26.432140 4339 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32892:
table { table_name: "default.Table_3cbad8a0189443d9b60b0abc4460862a" } modify_external_catalogs: true
I20250901 19:54:26.432730 4339 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32892:
table { table_name: "default.Table_3cbad8a0189443d9b60b0abc4460862a" } modify_external_catalogs: true
I20250901 19:54:26.433106 4337 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32892:
table { table_name: "default.Table_079afb395d5c4c17a18fd7e77548109d" } modify_external_catalogs: true
I20250901 19:54:26.433210 4336 master_service.cc:948] ReplaceTablet: received request to replace tablet 5f37af30f3894824b7ce48101e1fda8a from {username='slave'} at 127.0.0.1:32892
I20250901 19:54:26.433365 4337 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32892:
table { table_name: "default.Table_079afb395d5c4c17a18fd7e77548109d" } modify_external_catalogs: true
I20250901 19:54:26.433535 4334 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32892:
table { table_name: "default.Table_11ed1535329246fc866753e35e0cb6a1" } modify_external_catalogs: true
I20250901 19:54:26.432859 4338 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32892:
name: "default.Table_788e14a6980547248f5627f671a238a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:26.434007 4332 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32892:
name: "default.Table_8728509cfe374d5ea49110ea201c1659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:26.433408 4335 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32892:
name: "default.Table_edff33351bd64d16aec20db4230ac013"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:26.433663 4333 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32892:
name: "default.Table_69070232fcfc4853b5fff18e23fbf965"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:26.433835 4334 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32892:
table { table_name: "default.Table_11ed1535329246fc866753e35e0cb6a1" } modify_external_catalogs: true
I20250901 19:54:26.436887 4330 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:32892:
table { table_name: "default.Table_0a1ca85b493d4aa285d85b17df52c4aa" } new_table_name: "default.Table_ed1ee4c8517b4614aafb4f78827e4f6e" modify_external_catalogs: true
I20250901 19:54:26.437739 4331 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32892:
name: "default.Table_b7d788fb008945558d1e6fd4dde4f706"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:26.441617 4493 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Refusing update from remote peer 4c9163569ae1418090a245256e263186: Log matching property violated. Preceding OpId in replica: term: 3 index: 42. Preceding OpId from leader: term: 5 index: 44. (index mismatch)
F20250901 19:54:26.441708 4395 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_0a1ca85b493d4aa285d85b17df52c4aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_0a1ca85b493d4aa285d85b17df52c4aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0a1ca85b493d4aa285d85b17df52c4aa\",\"kudu.table_id\":\"fe4c5d5ccf384fbfbf05cb7b4f0a1bb6\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_c32254b687594ed89daf3edaded35200, message={"server":"","servicePrincipal":"","db":"default","table":"table_c32254b687594ed89daf3edaded35200","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c32254b687594ed89daf3edaded35200\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_c32254b687594ed89daf3edaded35200\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c32254b687594ed89daf3edaded35200\",\"kudu.table_id\":\"268741b95ddb403f8f59e4219723950e\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250901 19:54:26.442448 4543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 38, Time since last communication: 0.000s
I20250901 19:54:26.442646 4543 consensus_queue.cc:1230] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [LEADER]: Peer 1d944ed54f8c43c9b8c4e7d589dad395 log is divergent from this leader: its last log entry 3.42 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 38
I20250901 19:54:26.444083 4493 raft_consensus.cc:1215] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 2.38->[3.39-5.44] Dedup: 3.40->[4.41-5.44]
I20250901 19:54:26.444342 4493 pending_rounds.cc:85] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Aborting all ops after (but not including) 40
I20250901 19:54:26.444490 4493 pending_rounds.cc:107] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Aborting uncommitted WRITE_OP operation due to leader change: 3.41
@ 0x7f659488fdcd google::LogMessage::Fail() at ??:0
I20250901 19:54:26.445149 4493 pending_rounds.cc:107] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Aborting uncommitted WRITE_OP operation due to leader change: 3.42
@ 0x7f6594893b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f659488f7cc google::LogMessage::Flush() at ??:0
@ 0x7f659488fa11 google::LogMessage::~LogMessage() at ??:0
W20250901 19:54:26.451695 4322 consensus_peers.cc:489] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 -> Peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Couldn't send request to peer 08076b584d2a4aec8fd9868f85c9c7cd. Status: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f65aacc4794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250901 19:54:26.458647 4553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:26.459030 4553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:26.459486 4523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c9163569ae1418090a245256e263186. Latest consensus state: current_term: 5 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:26.459748 4523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:26.460096 4556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:26.460345 4556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:26.461202 4543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:26.461540 4543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:26.462349 4336 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Sending DeleteTablet for 3 replicas of tablet 5f37af30f3894824b7ce48101e1fda8a
I20250901 19:54:26.463754 4556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:26.463974 4556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:26.464396 4336 catalog_manager.cc:6652] ReplaceTablet: tablet 5f37af30f3894824b7ce48101e1fda8a deleted and replaced by tablet 7b163f22e81940f2a8d9249c5cb64afc
I20250901 19:54:26.464939 4336 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:32892:
table { table_name: "default.Table_5cf2893927ed42d597f83e0a04da2245" } new_table_name: "default.Table_3d911927cc9648b39bc8fae3c7d25c87" modify_external_catalogs: true
@ 0x7f65aacc3041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250901 19:54:26.475310 3616 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f37af30f3894824b7ce48101e1fda8a with delete_type TABLET_DATA_DELETED (replaced by tablet 7b163f22e81940f2a8d9249c5cb64afc) from {username='slave'} at 127.0.0.1:55266
I20250901 19:54:26.493273 3757 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f37af30f3894824b7ce48101e1fda8a with delete_type TABLET_DATA_DELETED (replaced by tablet 7b163f22e81940f2a8d9249c5cb64afc) from {username='slave'} at 127.0.0.1:50334
I20250901 19:54:26.494518 4563 tablet_replica.cc:331] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21: stopping tablet replica
I20250901 19:54:26.494843 4563 raft_consensus.cc:2241] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [term 1 LEADER]: Raft consensus shutting down.
I20250901 19:54:26.495460 4563 raft_consensus.cc:2270] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:26.499577 3895 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f37af30f3894824b7ce48101e1fda8a with delete_type TABLET_DATA_DELETED (replaced by tablet 7b163f22e81940f2a8d9249c5cb64afc) from {username='slave'} at 127.0.0.1:48766
I20250901 19:54:26.504268 4565 tablet_replica.cc:331] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734: stopping tablet replica
I20250901 19:54:26.504261 4563 ts_tablet_manager.cc:1905] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:26.504535 4565 raft_consensus.cc:2241] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:26.504691 4566 tablet_replica.cc:331] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8: stopping tablet replica
I20250901 19:54:26.504745 4565 raft_consensus.cc:2270] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:26.504917 4566 raft_consensus.cc:2241] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:26.505141 4566 raft_consensus.cc:2270] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:26.505362 4565 ts_tablet_manager.cc:1905] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:26.505770 4566 ts_tablet_manager.cc:1905] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f65aaccacbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250901 19:54:26.509965 3757 tablet_service.cc:1468] Processing CreateTablet for tablet 7b163f22e81940f2a8d9249c5cb64afc (DEFAULT_TABLE table=default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:26.510154 3616 tablet_service.cc:1468] Processing CreateTablet for tablet 7b163f22e81940f2a8d9249c5cb64afc (DEFAULT_TABLE table=default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:26.510282 3895 tablet_service.cc:1468] Processing CreateTablet for tablet 7b163f22e81940f2a8d9249c5cb64afc (DEFAULT_TABLE table=default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:26.510493 3757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b163f22e81940f2a8d9249c5cb64afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:26.510587 3616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b163f22e81940f2a8d9249c5cb64afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:26.510658 3895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b163f22e81940f2a8d9249c5cb64afc. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f65aaccab39 std::_Function_handler<>::_M_invoke() at ??:0
I20250901 19:54:26.518990 4565 ts_tablet_manager.cc:1918] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 19:54:26.519168 4565 log.cc:1199] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/wal/wals/5f37af30f3894824b7ce48101e1fda8a
I20250901 19:54:26.519802 4565 ts_tablet_manager.cc:1939] T 5f37af30f3894824b7ce48101e1fda8a P d397574faadb441192c99d530f14a734: Deleting consensus metadata
I20250901 19:54:26.521450 4323 catalog_manager.cc:4928] TS d397574faadb441192c99d530f14a734 (127.2.246.194:33557): tablet 5f37af30f3894824b7ce48101e1fda8a (table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]) successfully deleted
I20250901 19:54:26.521962 4566 ts_tablet_manager.cc:1918] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 19:54:26.522100 4566 log.cc:1199] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/wal/wals/5f37af30f3894824b7ce48101e1fda8a
I20250901 19:54:26.522568 4566 ts_tablet_manager.cc:1939] T 5f37af30f3894824b7ce48101e1fda8a P e0064a1b18e5470b85bde2277757aac8: Deleting consensus metadata
I20250901 19:54:26.523792 4322 catalog_manager.cc:4928] TS e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271): tablet 5f37af30f3894824b7ce48101e1fda8a (table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]) successfully deleted
I20250901 19:54:26.528245 4570 tablet_bootstrap.cc:492] T 7b163f22e81940f2a8d9249c5cb64afc P d397574faadb441192c99d530f14a734: Bootstrap starting.
@ 0xa774dd std::function<>::operator()() at ??:?
I20250901 19:54:26.530758 4570 tablet_bootstrap.cc:654] T 7b163f22e81940f2a8d9249c5cb64afc P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:26.539521 4563 ts_tablet_manager.cc:1918] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 19:54:26.539739 4563 log.cc:1199] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/wal/wals/5f37af30f3894824b7ce48101e1fda8a
I20250901 19:54:26.540342 4563 ts_tablet_manager.cc:1939] T 5f37af30f3894824b7ce48101e1fda8a P c5b48d93671f4eb8980759b876e9ca21: Deleting consensus metadata
I20250901 19:54:26.542062 4321 catalog_manager.cc:4928] TS c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): tablet 5f37af30f3894824b7ce48101e1fda8a (table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]) successfully deleted
I20250901 19:54:26.546739 4569 tablet_bootstrap.cc:492] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
@ 0x7f65975eccc9 kudu::Thread::SuperviseThread() at ??:0
I20250901 19:54:26.550099 4569 tablet_bootstrap.cc:654] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:26.552285 4568 tablet_bootstrap.cc:492] T 7b163f22e81940f2a8d9249c5cb64afc P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
@ 0x7f659aa0e6db start_thread at ??:0
I20250901 19:54:26.554795 4568 tablet_bootstrap.cc:654] T 7b163f22e81940f2a8d9249c5cb64afc P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:26.556133 4570 tablet_bootstrap.cc:492] T 7b163f22e81940f2a8d9249c5cb64afc P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:26.556293 4570 ts_tablet_manager.cc:1397] T 7b163f22e81940f2a8d9249c5cb64afc P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.009s
@ 0x7f659374071f clone at ??:0
I20250901 19:54:26.557682 4570 raft_consensus.cc:357] T 7b163f22e81940f2a8d9249c5cb64afc P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:26.557909 4570 raft_consensus.cc:383] T 7b163f22e81940f2a8d9249c5cb64afc P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:26.558002 4570 raft_consensus.cc:738] T 7b163f22e81940f2a8d9249c5cb64afc P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:26.558293 4570 consensus_queue.cc:260] T 7b163f22e81940f2a8d9249c5cb64afc P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:26.558836 3824 heartbeater.cc:499] Master 127.2.246.254:33523 was elected leader, sending a full tablet report...
I20250901 19:54:26.562167 4570 ts_tablet_manager.cc:1428] T 7b163f22e81940f2a8d9249c5cb64afc P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250901 19:54:26.563213 4569 tablet_bootstrap.cc:492] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:26.563378 4569 ts_tablet_manager.cc:1397] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20250901 19:54:26.564327 4569 raft_consensus.cc:357] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:26.564518 4569 raft_consensus.cc:383] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:26.564618 4569 raft_consensus.cc:738] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:26.564779 4568 tablet_bootstrap.cc:492] T 7b163f22e81940f2a8d9249c5cb64afc P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:26.564926 4568 ts_tablet_manager.cc:1397] T 7b163f22e81940f2a8d9249c5cb64afc P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20250901 19:54:26.564855 4569 consensus_queue.cc:260] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:26.565734 4568 raft_consensus.cc:357] T 7b163f22e81940f2a8d9249c5cb64afc P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:26.565920 4568 raft_consensus.cc:383] T 7b163f22e81940f2a8d9249c5cb64afc P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:26.566004 4568 raft_consensus.cc:738] T 7b163f22e81940f2a8d9249c5cb64afc P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:26.566274 4568 consensus_queue.cc:260] T 7b163f22e81940f2a8d9249c5cb64afc P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:26.566852 4568 ts_tablet_manager.cc:1428] T 7b163f22e81940f2a8d9249c5cb64afc P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 19:54:26.567377 4569 ts_tablet_manager.cc:1428] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250901 19:54:26.591380 4273 raft_consensus.cc:491] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:26.591588 4273 raft_consensus.cc:513] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:26.592325 4273 leader_election.cc:290] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:26.592924 3777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b163f22e81940f2a8d9249c5cb64afc" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734" is_pre_election: true
I20250901 19:54:26.592985 3637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b163f22e81940f2a8d9249c5cb64afc" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21" is_pre_election: true
I20250901 19:54:26.593161 3777 raft_consensus.cc:2466] T 7b163f22e81940f2a8d9249c5cb64afc P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 0.
I20250901 19:54:26.593256 3637 raft_consensus.cc:2466] T 7b163f22e81940f2a8d9249c5cb64afc P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 0.
I20250901 19:54:26.593727 3851 leader_election.cc:304] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734, e0064a1b18e5470b85bde2277757aac8; no voters:
I20250901 19:54:26.594017 4273 raft_consensus.cc:2802] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:26.594137 4273 raft_consensus.cc:491] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:26.594238 4273 raft_consensus.cc:3058] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:26.596428 4273 raft_consensus.cc:513] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:26.597105 4273 leader_election.cc:290] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 election: Requested vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:26.597682 3637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b163f22e81940f2a8d9249c5cb64afc" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21"
I20250901 19:54:26.597738 3777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b163f22e81940f2a8d9249c5cb64afc" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734"
I20250901 19:54:26.597905 3637 raft_consensus.cc:3058] T 7b163f22e81940f2a8d9249c5cb64afc P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:26.598008 3777 raft_consensus.cc:3058] T 7b163f22e81940f2a8d9249c5cb64afc P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:26.600231 3637 raft_consensus.cc:2466] T 7b163f22e81940f2a8d9249c5cb64afc P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 1.
I20250901 19:54:26.600745 3849 leader_election.cc:304] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, e0064a1b18e5470b85bde2277757aac8; no voters:
I20250901 19:54:26.600844 3777 raft_consensus.cc:2466] T 7b163f22e81940f2a8d9249c5cb64afc P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 1.
I20250901 19:54:26.601110 4273 raft_consensus.cc:2802] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:26.601609 4273 raft_consensus.cc:695] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [term 1 LEADER]: Becoming Leader. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Running, Role: LEADER
I20250901 19:54:26.602008 4273 consensus_queue.cc:237] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
W20250901 19:54:26.741830 3851 connection.cc:537] client connection to 127.2.246.254:33523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:26.742277 3573 connection.cc:537] client connection to 127.2.246.254:33523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:26.742709 3713 connection.cc:537] client connection to 127.2.246.254:33523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:26.742936 3973 connection.cc:537] client connection to 127.2.246.254:33523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:26.743615 3973 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.246.254:33523)
W20250901 19:54:26.743937 3973 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.246.254:33523)
W20250901 19:54:26.744163 3973 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.246.254:33523)
W20250901 19:54:26.745422 3973 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.246.254:33523)
I20250901 19:54:26.976732 4584 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:54:26.976961 4584 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:26.978813 4584 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:26.979813 4469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:26.981217 4468 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
W20250901 19:54:26.981478 4469 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
I20250901 19:54:26.981642 4469 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186
I20250901 19:54:26.981990 4584 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250901 19:54:27.396703 4273 consensus_queue.cc:1035] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:27.402706 4593 consensus_queue.cc:1035] T 7b163f22e81940f2a8d9249c5cb64afc P e0064a1b18e5470b85bde2277757aac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:27.630920 4598 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:54:27.631120 4598 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:27.631932 4598 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:27.633805 4469 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:27.633994 4468 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:27.634126 4468 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186
I20250901 19:54:27.634377 4598 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250901 19:54:28.525701 4613 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:54:28.525982 4613 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:28.526636 4613 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:28.529033 4468 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
W20250901 19:54:28.529213 4469 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
I20250901 19:54:28.529345 4469 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186
I20250901 19:54:28.529611 4613 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250901 19:54:28.920966 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.252:37003
--webserver_interface=127.2.246.252
--webserver_port=36357
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250901 19:54:29.268047 4624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:29.268397 4624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:29.268533 4624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 19:54:29.276954 4624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 19:54:29.277060 4624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:29.277123 4624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 19:54:29.277171 4624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 19:54:29.288350 4624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.252:37003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.246.252
--webserver_port=36357
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:29.290017 4624 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:29.291939 4624 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 19:54:29.300222 4631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:29.300187 4632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:29.300732 4634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:29.301869 4624 server_base.cc:1047] running on GCE node
I20250901 19:54:29.302645 4624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:29.304230 4624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:29.305612 4624 hybrid_clock.cc:648] HybridClock initialized: now 1756756469305551 us; error 36 us; skew 500 ppm
I20250901 19:54:29.306012 4624 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:29.308374 4624 webserver.cc:480] Webserver started at http://127.2.246.252:36357/ using document root <none> and password file <none>
I20250901 19:54:29.309142 4624 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:29.309240 4624 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:29.315331 4624 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20250901 19:54:29.318637 4640 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:29.320268 4624 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250901 19:54:29.320489 4624 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
uuid: "08076b584d2a4aec8fd9868f85c9c7cd"
format_stamp: "Formatted at 2025-09-01 19:54:14 on dist-test-slave-jnn4"
I20250901 19:54:29.321543 4624 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:29.348300 4624 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:29.349473 4624 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:29.349809 4624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:29.370113 4624 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.252:37003
I20250901 19:54:29.370131 4691 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.252:37003 every 8 connection(s)
I20250901 19:54:29.371994 4624 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
I20250901 19:54:29.372493 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 4624
I20250901 19:54:29.379895 4692 sys_catalog.cc:263] Verifying existing consensus state
I20250901 19:54:29.384321 4692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Bootstrap starting.
I20250901 19:54:29.397091 4692 log.cc:826] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:29.428488 3683 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
I20250901 19:54:29.437842 3823 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
I20250901 19:54:29.438788 4692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=42 ignored=0} inserts{seen=25 ignored=0} mutations{seen=27 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 19:54:29.439503 4692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Bootstrap complete.
I20250901 19:54:29.445204 4692 raft_consensus.cc:357] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:29.445962 4692 raft_consensus.cc:738] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08076b584d2a4aec8fd9868f85c9c7cd, State: Initialized, Role: FOLLOWER
I20250901 19:54:29.446592 4692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:29.447368 4701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:29.447711 4701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:29.448482 4692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: configured and running, proceeding with master startup.
I20250901 19:54:29.462541 4715 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:29.462788 4715 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: loading cluster ID for follower catalog manager: success
I20250901 19:54:29.458441 4624 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:54:29.464661 4624 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:54:29.467459 4715 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: acquiring CA information for follower catalog manager: success
I20250901 19:54:29.470623 4715 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 19:54:29.790140 4722 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:54:29.790522 4722 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:29.791811 4722 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
I20250901 19:54:29.795585 3961 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
W20250901 19:54:29.797288 4469 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
I20250901 19:54:29.798782 4657 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" instance_seqno: 1756756457020957) as {username='slave'} at 127.2.246.195:49715; Asking this server to re-register.
I20250901 19:54:29.799718 3961 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:29.801990 4657 ts_manager.cc:194] Registered new tserver with Master: e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:29.803336 4667 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" candidate_term: 6 candidate_status { last_received { term: 5 index: 45 } } ignore_live_leader: false dest_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" is_pre_election: true
I20250901 19:54:29.803910 4667 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d944ed54f8c43c9b8c4e7d589dad395 in term 4.
I20250901 19:54:29.804489 4468 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 4c9163569ae1418090a245256e263186
I20250901 19:54:29.804811 4722 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250901 19:54:29.804945 4722 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:54:29.805071 4722 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 5 FOLLOWER]: Advancing to term 6
I20250901 19:54:29.807054 4722 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:29.807734 4722 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 election: Requested vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
I20250901 19:54:29.808610 4667 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" candidate_term: 6 candidate_status { last_received { term: 5 index: 45 } } ignore_live_leader: false dest_uuid: "08076b584d2a4aec8fd9868f85c9c7cd"
I20250901 19:54:29.808806 4667 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 4 FOLLOWER]: Advancing to term 6
W20250901 19:54:29.812955 4469 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
I20250901 19:54:29.813228 4667 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d944ed54f8c43c9b8c4e7d589dad395 in term 6.
I20250901 19:54:29.813720 4468 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 4c9163569ae1418090a245256e263186
I20250901 19:54:29.814194 4722 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 6 FOLLOWER]: Leader election won for term 6
I20250901 19:54:29.814708 4722 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 6 LEADER]: Becoming Leader. State: Replica: 1d944ed54f8c43c9b8c4e7d589dad395, State: Running, Role: LEADER
I20250901 19:54:29.815330 4722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 5.45, Last appended by leader: 45, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:29.817277 4726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d944ed54f8c43c9b8c4e7d589dad395. Latest consensus state: current_term: 6 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:29.817577 4726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:29.819141 4728 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 19:54:29.822602 4728 catalog_manager.cc:671] Loaded metadata for table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077]
I20250901 19:54:29.823150 4728 catalog_manager.cc:671] Loaded metadata for table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]
I20250901 19:54:29.823599 4728 catalog_manager.cc:671] Loaded metadata for table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]
I20250901 19:54:29.824046 4728 catalog_manager.cc:671] Loaded metadata for table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67]
I20250901 19:54:29.824461 4728 catalog_manager.cc:671] Loaded metadata for table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]
I20250901 19:54:29.824884 4728 catalog_manager.cc:671] Loaded metadata for table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]
I20250901 19:54:29.825258 4728 catalog_manager.cc:671] Loaded metadata for table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8]
I20250901 19:54:29.825657 4728 catalog_manager.cc:671] Loaded metadata for table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616]
I20250901 19:54:29.826023 4728 catalog_manager.cc:671] Loaded metadata for table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]
I20250901 19:54:29.829270 4728 tablet_loader.cc:96] loaded metadata for tablet 1d21a96219ab4f5f96bc6d9976f990ff (table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896])
I20250901 19:54:29.830057 4728 tablet_loader.cc:96] loaded metadata for tablet 7ab5ab06eece48fdb346f232247f72e0 (table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077])
I20250901 19:54:29.830413 4728 tablet_loader.cc:96] loaded metadata for tablet 7b163f22e81940f2a8d9249c5cb64afc (table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59])
I20250901 19:54:29.830780 4728 tablet_loader.cc:96] loaded metadata for tablet a5067dd498ef4331bd9d9f3bcbcaec44 (table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612])
I20250901 19:54:29.831185 4728 tablet_loader.cc:96] loaded metadata for tablet b867925c426149c99ffdd6d5df396462 (table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6])
I20250901 19:54:29.831612 4728 tablet_loader.cc:96] loaded metadata for tablet e003ca1913284f239f5b0474fc621abe (table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8])
I20250901 19:54:29.832003 4728 tablet_loader.cc:96] loaded metadata for tablet e1d516f8388c4a099f81d6e5a3c7cf4d (table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3])
I20250901 19:54:29.832414 4728 tablet_loader.cc:96] loaded metadata for tablet e43433b1ce9646e49962fe44bbed2491 (table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67])
I20250901 19:54:29.832798 4728 tablet_loader.cc:96] loaded metadata for tablet f4fe9cadf6414b548cbd95572d9ed1c8 (table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616])
I20250901 19:54:29.833250 4728 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 19:54:29.834427 4728 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:29.834543 4728 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 19:54:29.836126 4728 catalog_manager.cc:1506] Loading token signing keys...
I20250901 19:54:29.837109 4728 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Loaded TSK: 0
I20250901 19:54:29.838011 4728 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 19:54:29.838950 4728 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 19:54:29.839916 4728 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250901 19:54:29.864295 4482 master_service.cc:948] ReplaceTablet: received request to replace tablet 7b163f22e81940f2a8d9249c5cb64afc from {username='slave'} at 127.0.0.1:53954
I20250901 19:54:29.864740 4483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53954:
table { table_name: "default.Table_51c52d3eec714bbd9a77bed453c02bb1" } modify_external_catalogs: true
I20250901 19:54:29.865216 4476 master_service.cc:948] ReplaceTablet: received request to replace tablet 5f37af30f3894824b7ce48101e1fda8a from {username='slave'} at 127.0.0.1:53954
I20250901 19:54:29.865020 4481 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53954:
name: "default.Table_240cfd071a7c475bb44ff628d903bf9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:29.866009 4479 master_service.cc:948] ReplaceTablet: received request to replace tablet 516ed4484b174a8d8cb8c6b9f076d31e from {username='slave'} at 127.0.0.1:53954
I20250901 19:54:29.866110 4476 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53954:
name: "default.Table_edff33351bd64d16aec20db4230ac013"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:29.865047 4483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53954:
table { table_name: "default.Table_51c52d3eec714bbd9a77bed453c02bb1" } modify_external_catalogs: true
I20250901 19:54:29.866425 4480 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53954:
name: "default.Table_14959779c0504cb4bf46faa666221361"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:29.866710 4477 master_service.cc:948] ReplaceTablet: received request to replace tablet e43433b1ce9646e49962fe44bbed2491 from {username='slave'} at 127.0.0.1:53954
I20250901 19:54:29.866815 4479 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53954:
table { table_name: "default.Table_079afb395d5c4c17a18fd7e77548109d" } modify_external_catalogs: true
I20250901 19:54:29.867030 4478 master_service.cc:948] ReplaceTablet: received request to replace tablet e43433b1ce9646e49962fe44bbed2491 from {username='slave'} at 127.0.0.1:53954
I20250901 19:54:29.867107 4479 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53954:
table { table_name: "default.Table_079afb395d5c4c17a18fd7e77548109d" } modify_external_catalogs: true
I20250901 19:54:29.866792 4474 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53954:
table { table_name: "default.Table_11ed1535329246fc866753e35e0cb6a1" } modify_external_catalogs: true
I20250901 19:54:29.867483 4474 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53954:
table { table_name: "default.Table_11ed1535329246fc866753e35e0cb6a1" } modify_external_catalogs: true
I20250901 19:54:29.867506 4475 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53954:
name: "default.Table_4c54782dd2bb4c66a3be449970979d0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:29.872725 4667 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Refusing update from remote peer 1d944ed54f8c43c9b8c4e7d589dad395: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 6 index: 47. (index mismatch)
I20250901 19:54:29.873617 4722 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 42, Time since last communication: 0.000s
W20250901 19:54:29.873965 4469 consensus_peers.cc:489] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 -> Peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Couldn't send request to peer 4c9163569ae1418090a245256e263186. Status: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250901 19:54:29.875029 4536 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_0a1ca85b493d4aa285d85b17df52c4aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_0a1ca85b493d4aa285d85b17df52c4aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0a1ca85b493d4aa285d85b17df52c4aa\",\"kudu.table_id\":\"fe4c5d5ccf384fbfbf05cb7b4f0a1bb6\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_c32254b687594ed89daf3edaded35200, message={"server":"","servicePrincipal":"","db":"default","table":"table_c32254b687594ed89daf3edaded35200","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c32254b687594ed89daf3edaded35200\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_c32254b687594ed89daf3edaded35200\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c32254b687594ed89daf3edaded35200\",\"kudu.table_id\":\"268741b95ddb403f8f59e4219723950e\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fa462a1edcd google::LogMessage::Fail() at ??:0
@ 0x7fa462a22b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa462a1e7cc google::LogMessage::Flush() at ??:0
@ 0x7fa462a1ea11 google::LogMessage::~LogMessage() at ??:0
I20250901 19:54:29.883059 4701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d944ed54f8c43c9b8c4e7d589dad395. Latest consensus state: current_term: 6 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:29.883359 4701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:29.883812 4733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:29.884130 4733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:29.885643 4730 mvcc.cc:204] Tried to move back new op lower bound from 7195674486658281472 to 7195674486155137024. Current Snapshot: MvccSnapshot[applied={T|T < 7195674486658281472 or (T in {7195674486658281472})}]
I20250901 19:54:29.885780 4729 mvcc.cc:204] Tried to move back new op lower bound from 7195674500588220416 to 7195674500367917056. Current Snapshot: MvccSnapshot[applied={T|T < 7195674500588220416}]
I20250901 19:54:29.887764 4733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:29.887987 4733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:29.890285 4477 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Sending DeleteTablet for 3 replicas of tablet e43433b1ce9646e49962fe44bbed2491
I20250901 19:54:29.890916 4482 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Sending DeleteTablet for 3 replicas of tablet 7b163f22e81940f2a8d9249c5cb64afc
@ 0x7fa478e53794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250901 19:54:29.893592 4482 catalog_manager.cc:6652] ReplaceTablet: tablet 7b163f22e81940f2a8d9249c5cb64afc deleted and replaced by tablet e0063b69ff9a48f6a1229cd26704c66d
I20250901 19:54:29.895393 4477 catalog_manager.cc:6652] ReplaceTablet: tablet e43433b1ce9646e49962fe44bbed2491 deleted and replaced by tablet 5d8e6708fad646d39f3cd20381c1c8aa
@ 0x7fa478e52041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fa478e59cbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fa478e59b39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250901 19:54:29.911855 4482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53954:
name: "default.Table_788e14a6980547248f5627f671a238a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:29.912541 4478 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53954:
table { table_name: "default.Table_3cbad8a0189443d9b60b0abc4460862a" } modify_external_catalogs: true
I20250901 19:54:29.912786 4478 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53954:
table { table_name: "default.Table_3cbad8a0189443d9b60b0abc4460862a" } modify_external_catalogs: true
I20250901 19:54:29.914542 4726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:29.914137 4477 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53954:
name: "default.Table_69070232fcfc4853b5fff18e23fbf965"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fa46577bcc9 kudu::Thread::SuperviseThread() at ??:0
I20250901 19:54:29.914769 4726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: LEADER
@ 0x7fa468b9d6db start_thread at ??:0
@ 0x7fa4618cf71f clone at ??:0
I20250901 19:54:29.935132 4657 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" instance_seqno: 1756756455802456) as {username='slave'} at 127.2.246.193:44985; Asking this server to re-register.
I20250901 19:54:29.936074 3683 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:29.937975 4657 ts_manager.cc:194] Registered new tserver with Master: c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:29.941699 4657 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d397574faadb441192c99d530f14a734" instance_seqno: 1756756456460392) as {username='slave'} at 127.2.246.194:45825; Asking this server to re-register.
I20250901 19:54:29.942428 3823 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:29.944365 4657 ts_manager.cc:194] Registered new tserver with Master: d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
W20250901 19:54:30.109078 3713 connection.cc:537] client connection to 127.2.246.253:34127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:30.110163 4741 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 19:54:29.893138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.2.246.194:33557)
0901 19:54:29.922254 (+ 29116us) server_negotiation.cc:206] Beginning negotiation
0901 19:54:29.922263 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0901 19:54:29.922300 (+ 37us) server_negotiation.cc:407] Connection header received
0901 19:54:29.922388 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 19:54:29.922394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 19:54:29.922480 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 19:54:29.922777 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 19:54:30.109549 (+186772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50346: BlockingRecv error: recv got EOF from 127.0.0.1:50346 (error 108)
Metrics: {"server-negotiator.queue_time_us":28950,"thread_start_us":28845,"threads_started":1}
W20250901 19:54:30.110337 4743 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 19:54:29.893718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.2.246.193:37767)
0901 19:54:29.920036 (+ 26318us) server_negotiation.cc:206] Beginning negotiation
0901 19:54:29.920046 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0901 19:54:29.920073 (+ 27us) server_negotiation.cc:407] Connection header received
0901 19:54:29.920156 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 19:54:29.920166 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 19:54:29.920264 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 19:54:29.921552 (+ 1288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 19:54:29.922951 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 19:54:29.924686 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 19:54:30.109164 (+184478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55282: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":26157,"thread_start_us":26607,"threads_started":1}
W20250901 19:54:30.110651 3973 connection.cc:537] client connection to 127.2.246.253:34127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:30.110997 4742 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 19:54:29.893431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.2.246.195:33271)
0901 19:54:29.915852 (+ 22421us) server_negotiation.cc:206] Beginning negotiation
0901 19:54:29.915863 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0901 19:54:30.110139 (+194276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48774: BlockingRecv error: recv got EOF from 127.0.0.1:48774 (error 108)
Metrics: {"server-negotiator.queue_time_us":22252,"thread_start_us":22109,"threads_started":1}
W20250901 19:54:30.111424 3973 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.246.253:34127)
W20250901 19:54:30.111928 3973 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.246.253:34127)
W20250901 19:54:30.112124 3973 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.246.253:34127)
W20250901 19:54:30.112746 3973 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.246.253:34127)
I20250901 19:54:30.387272 4733 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:30.387508 4733 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:30.389299 4733 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:30.390157 4643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:30.391834 4643 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:30.392066 4643 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:30.392197 4643 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:30.392532 4733 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250901 19:54:30.990270 4762 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:30.990516 4762 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:30.991271 4762 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:30.995512 4643 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:30.995746 4643 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:30.995908 4643 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:30.996194 4762 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250901 19:54:31.600294 4768 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:31.600476 4768 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:31.601073 4768 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:31.603538 4643 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:31.603802 4643 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:31.603895 4643 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:31.604219 4768 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250901 19:54:32.638185 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.253:34127
--webserver_interface=127.2.246.253
--webserver_port=36885
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250901 19:54:32.825660 4778 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:32.825932 4778 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:32.826589 4778 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:32.828902 4643 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:32.829118 4643 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:32.829242 4643 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:32.829494 4778 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250901 19:54:32.986035 4776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:32.986429 4776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:32.986559 4776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 19:54:32.995071 4776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 19:54:32.995198 4776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:32.995287 4776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 19:54:32.995357 4776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 19:54:33.005314 4776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.253:34127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.246.253
--webserver_port=36885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:33.007088 4776 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:33.009018 4776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 19:54:33.017019 4785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:33.018481 4786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:33.018512 4788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:33.019614 4776 server_base.cc:1047] running on GCE node
I20250901 19:54:33.020514 4776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:33.021977 4776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:33.023274 4776 hybrid_clock.cc:648] HybridClock initialized: now 1756756473023220 us; error 52 us; skew 500 ppm
I20250901 19:54:33.023677 4776 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:33.026119 4776 webserver.cc:480] Webserver started at http://127.2.246.253:36885/ using document root <none> and password file <none>
I20250901 19:54:33.026983 4776 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:33.027096 4776 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:33.032810 4776 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20250901 19:54:33.036175 4794 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:33.037847 4776 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250901 19:54:33.038053 4776 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
uuid: "1d944ed54f8c43c9b8c4e7d589dad395"
format_stamp: "Formatted at 2025-09-01 19:54:14 on dist-test-slave-jnn4"
I20250901 19:54:33.038776 4776 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:33.084339 4776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:33.085265 4776 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:33.085604 4776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:33.106335 4776 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.253:34127
I20250901 19:54:33.106356 4845 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.253:34127 every 8 connection(s)
I20250901 19:54:33.108243 4776 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/info.pb
I20250901 19:54:33.110071 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 4776
I20250901 19:54:33.116894 4846 sys_catalog.cc:263] Verifying existing consensus state
I20250901 19:54:33.124053 4846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Bootstrap starting.
I20250901 19:54:33.138345 3685 heartbeater.cc:344] Connected to a master server at 127.2.246.253:34127
I20250901 19:54:33.141320 3829 heartbeater.cc:344] Connected to a master server at 127.2.246.253:34127
I20250901 19:54:33.143869 4846 log.cc:826] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:33.185254 4846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Bootstrap replayed 1/1 log segments. Stats: ops{read=51 overwritten=2 applied=48 ignored=0} inserts{seen=28 ignored=0} mutations{seen=31 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250901 19:54:33.186055 4846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Bootstrap complete.
I20250901 19:54:33.191861 4846 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:33.194586 4846 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d944ed54f8c43c9b8c4e7d589dad395, State: Initialized, Role: FOLLOWER
I20250901 19:54:33.195385 4846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 48, Last appended: 6.49, Last appended by leader: 49, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:33.196218 4853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:33.196473 4853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:33.197364 4846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 19:54:33.210664 4867 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:33.210822 4867 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: loading cluster ID for follower catalog manager: success
I20250901 19:54:33.217114 4867 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: acquiring CA information for follower catalog manager: success
I20250901 19:54:33.221526 4867 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 19:54:33.222177 4776 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:54:33.226428 4776 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:54:33.308789 3963 heartbeater.cc:344] Connected to a master server at 127.2.246.253:34127
I20250901 19:54:33.311260 4811 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" instance_seqno: 1756756457020957) as {username='slave'} at 127.2.246.195:59679; Asking this server to re-register.
I20250901 19:54:33.312124 3963 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:33.314455 4811 ts_manager.cc:194] Registered new tserver with Master: e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:33.642545 4811 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" instance_seqno: 1756756455802456) as {username='slave'} at 127.2.246.193:39601; Asking this server to re-register.
I20250901 19:54:33.643278 3685 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:33.644364 4811 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d397574faadb441192c99d530f14a734" instance_seqno: 1756756456460392) as {username='slave'} at 127.2.246.194:47585; Asking this server to re-register.
I20250901 19:54:33.645305 3829 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:33.645458 4810 ts_manager.cc:194] Registered new tserver with Master: c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:33.646904 4810 ts_manager.cc:194] Registered new tserver with Master: d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:33.795018 4873 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:33.795307 4873 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:33.797250 4873 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:33.798713 4797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:33.800522 4797 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
I20250901 19:54:33.804443 4667 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" candidate_term: 7 candidate_status { last_received { term: 6 index: 49 } } ignore_live_leader: false dest_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" is_pre_election: true
I20250901 19:54:33.804701 4667 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d944ed54f8c43c9b8c4e7d589dad395 in term 6.
I20250901 19:54:33.805428 4796 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 4c9163569ae1418090a245256e263186
I20250901 19:54:33.805855 4873 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250901 19:54:33.805995 4873 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:33.806097 4873 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 6 FOLLOWER]: Advancing to term 7
I20250901 19:54:33.809703 4873 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:33.810312 4873 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 7 election: Requested vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
I20250901 19:54:33.810971 4667 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" candidate_term: 7 candidate_status { last_received { term: 6 index: 49 } } ignore_live_leader: false dest_uuid: "08076b584d2a4aec8fd9868f85c9c7cd"
I20250901 19:54:33.811146 4667 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 6 FOLLOWER]: Advancing to term 7
I20250901 19:54:33.813195 4667 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d944ed54f8c43c9b8c4e7d589dad395 in term 7.
I20250901 19:54:33.813640 4796 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395; no voters:
I20250901 19:54:33.813938 4873 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 7 FOLLOWER]: Leader election won for term 7
I20250901 19:54:33.814317 4873 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 7 LEADER]: Becoming Leader. State: Replica: 1d944ed54f8c43c9b8c4e7d589dad395, State: Running, Role: LEADER
I20250901 19:54:33.814788 4873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 6.49, Last appended by leader: 49, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:33.818280 4878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d944ed54f8c43c9b8c4e7d589dad395. Latest consensus state: current_term: 7 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:33.818518 4878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:34.282222 4667 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Refusing update from remote peer 1d944ed54f8c43c9b8c4e7d589dad395: Log matching property violated. Preceding OpId in replica: term: 6 index: 48. Preceding OpId from leader: term: 7 index: 50. (index mismatch)
I20250901 19:54:34.283128 4878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 48, Time since last communication: 0.000s
I20250901 19:54:34.287361 4883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d944ed54f8c43c9b8c4e7d589dad395. Latest consensus state: current_term: 7 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:34.287688 4883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:34.289155 4873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:34.289445 4873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:34.290114 4883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:34.290294 4883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:34.335676 4880 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 19:54:34.338430 4880 catalog_manager.cc:671] Loaded metadata for table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077]
I20250901 19:54:34.338922 4880 catalog_manager.cc:671] Loaded metadata for table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]
I20250901 19:54:34.339505 4880 catalog_manager.cc:671] Loaded metadata for table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]
I20250901 19:54:34.339972 4880 catalog_manager.cc:671] Loaded metadata for table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67]
I20250901 19:54:34.340333 4880 catalog_manager.cc:671] Loaded metadata for table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]
I20250901 19:54:34.340683 4880 catalog_manager.cc:671] Loaded metadata for table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]
I20250901 19:54:34.341024 4880 catalog_manager.cc:671] Loaded metadata for table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8]
I20250901 19:54:34.341359 4880 catalog_manager.cc:671] Loaded metadata for table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616]
I20250901 19:54:34.341742 4880 catalog_manager.cc:671] Loaded metadata for table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]
I20250901 19:54:34.344048 4880 tablet_loader.cc:96] loaded metadata for tablet 1d21a96219ab4f5f96bc6d9976f990ff (table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896])
I20250901 19:54:34.344511 4880 tablet_loader.cc:96] loaded metadata for tablet 5d8e6708fad646d39f3cd20381c1c8aa (table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67])
I20250901 19:54:34.345052 4880 tablet_loader.cc:96] loaded metadata for tablet 7ab5ab06eece48fdb346f232247f72e0 (table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077])
I20250901 19:54:34.345521 4880 tablet_loader.cc:96] loaded metadata for tablet a5067dd498ef4331bd9d9f3bcbcaec44 (table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612])
I20250901 19:54:34.345829 4880 tablet_loader.cc:96] loaded metadata for tablet b867925c426149c99ffdd6d5df396462 (table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6])
I20250901 19:54:34.346141 4880 tablet_loader.cc:96] loaded metadata for tablet e003ca1913284f239f5b0474fc621abe (table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8])
I20250901 19:54:34.346446 4880 tablet_loader.cc:96] loaded metadata for tablet e0063b69ff9a48f6a1229cd26704c66d (table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59])
I20250901 19:54:34.346715 4880 tablet_loader.cc:96] loaded metadata for tablet e1d516f8388c4a099f81d6e5a3c7cf4d (table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3])
I20250901 19:54:34.347116 4880 tablet_loader.cc:96] loaded metadata for tablet f4fe9cadf6414b548cbd95572d9ed1c8 (table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616])
I20250901 19:54:34.347483 4880 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 19:54:34.348587 4880 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:34.348685 4880 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 19:54:34.350093 4880 catalog_manager.cc:1506] Loading token signing keys...
I20250901 19:54:34.351034 4880 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Loaded TSK: 0
I20250901 19:54:34.352066 4880 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 19:54:34.353015 4880 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 19:54:34.353873 4880 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250901 19:54:34.385468 4810 master_service.cc:948] ReplaceTablet: received request to replace tablet e43433b1ce9646e49962fe44bbed2491 from {username='slave'} at 127.0.0.1:38242
I20250901 19:54:34.385556 4811 master_service.cc:948] ReplaceTablet: received request to replace tablet a5067dd498ef4331bd9d9f3bcbcaec44 from {username='slave'} at 127.0.0.1:38242
I20250901 19:54:34.386170 4809 master_service.cc:948] ReplaceTablet: received request to replace tablet a5067dd498ef4331bd9d9f3bcbcaec44 from {username='slave'} at 127.0.0.1:38242
I20250901 19:54:34.386247 4807 master_service.cc:948] ReplaceTablet: received request to replace tablet a5067dd498ef4331bd9d9f3bcbcaec44 from {username='slave'} at 127.0.0.1:38242
I20250901 19:54:34.386632 4808 master_service.cc:948] ReplaceTablet: received request to replace tablet 1d21a96219ab4f5f96bc6d9976f990ff from {username='slave'} at 127.0.0.1:38242
I20250901 19:54:34.387064 4806 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38242:
table { table_name: "default.Table_69ed2a785efc4a4f8077c8aa2ef89c35" } new_table_name: "default.Table_6d9fb4a3f26048cb95788236b2d5ec7f" modify_external_catalogs: true
I20250901 19:54:34.387239 4805 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38242:
name: "default.Table_b679bfff7a2f4d13b5e318b2ec42fc36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:34.388572 4810 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38242:
name: "default.Table_6f60f64babef49459553b6e490e198e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:34.388545 4804 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38242:
table { table_name: "default.Table_ea3686ff3ba94b9a984657e263ca0832" } new_table_name: "default.Table_f79692d68cd04e74ae3e3c67bc4f99a3" modify_external_catalogs: true
I20250901 19:54:34.389364 4803 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38242:
table { table_name: "default.Table_d29d54fc138b40009682d5b2585d25bd" } new_table_name: "default.Table_6a818be47b2e41e48156df825fbe0cc3" modify_external_catalogs: true
I20250901 19:54:34.390141 4802 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38242:
table { table_name: "default.Table_5cf2893927ed42d597f83e0a04da2245" } new_table_name: "default.Table_3d911927cc9648b39bc8fae3c7d25c87" modify_external_catalogs: true
F20250901 19:54:34.397236 4866 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_0a1ca85b493d4aa285d85b17df52c4aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_0a1ca85b493d4aa285d85b17df52c4aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0a1ca85b493d4aa285d85b17df52c4aa\",\"kudu.table_id\":\"fe4c5d5ccf384fbfbf05cb7b4f0a1bb6\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_c32254b687594ed89daf3edaded35200, message={"server":"","servicePrincipal":"","db":"default","table":"table_c32254b687594ed89daf3edaded35200","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c32254b687594ed89daf3edaded35200\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_c32254b687594ed89daf3edaded35200\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c32254b687594ed89daf3edaded35200\",\"kudu.table_id\":\"268741b95ddb403f8f59e4219723950e\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fe020444dcd google::LogMessage::Fail() at ??:0
I20250901 19:54:34.400535 4811 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Sending DeleteTablet for 3 replicas of tablet a5067dd498ef4331bd9d9f3bcbcaec44
@ 0x7fe020448b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe0204447cc google::LogMessage::Flush() at ??:0
@ 0x7fe020444a11 google::LogMessage::~LogMessage() at ??:0
I20250901 19:54:34.415562 4811 catalog_manager.cc:6652] ReplaceTablet: tablet a5067dd498ef4331bd9d9f3bcbcaec44 deleted and replaced by tablet 6579c1fbcfc04dac9345a8157b352591
I20250901 19:54:34.427729 4808 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Sending DeleteTablet for 3 replicas of tablet 1d21a96219ab4f5f96bc6d9976f990ff
I20250901 19:54:34.428364 4808 catalog_manager.cc:6652] ReplaceTablet: tablet 1d21a96219ab4f5f96bc6d9976f990ff deleted and replaced by tablet 957e86dd3dd940fabcab40043e801e6b
I20250901 19:54:34.444250 3616 tablet_service.cc:1515] Processing DeleteTablet for tablet a5067dd498ef4331bd9d9f3bcbcaec44 with delete_type TABLET_DATA_DELETED (replaced by tablet 6579c1fbcfc04dac9345a8157b352591) from {username='slave'} at 127.0.0.1:32952
I20250901 19:54:34.444998 4902 tablet_replica.cc:331] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21: stopping tablet replica
I20250901 19:54:34.445071 3616 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d21a96219ab4f5f96bc6d9976f990ff with delete_type TABLET_DATA_DELETED (replaced by tablet 957e86dd3dd940fabcab40043e801e6b) from {username='slave'} at 127.0.0.1:32952
I20250901 19:54:34.445222 4902 raft_consensus.cc:2241] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:34.446653 4902 raft_consensus.cc:2270] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7fe036879794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250901 19:54:34.447443 4902 ts_tablet_manager.cc:1905] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:34.451829 3758 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d21a96219ab4f5f96bc6d9976f990ff with delete_type TABLET_DATA_DELETED (replaced by tablet 957e86dd3dd940fabcab40043e801e6b) from {username='slave'} at 127.0.0.1:40758
I20250901 19:54:34.451846 3757 tablet_service.cc:1515] Processing DeleteTablet for tablet a5067dd498ef4331bd9d9f3bcbcaec44 with delete_type TABLET_DATA_DELETED (replaced by tablet 6579c1fbcfc04dac9345a8157b352591) from {username='slave'} at 127.0.0.1:40758
I20250901 19:54:34.453820 3895 tablet_service.cc:1515] Processing DeleteTablet for tablet a5067dd498ef4331bd9d9f3bcbcaec44 with delete_type TABLET_DATA_DELETED (replaced by tablet 6579c1fbcfc04dac9345a8157b352591) from {username='slave'} at 127.0.0.1:34696
I20250901 19:54:34.453991 3896 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d21a96219ab4f5f96bc6d9976f990ff with delete_type TABLET_DATA_DELETED (replaced by tablet 957e86dd3dd940fabcab40043e801e6b) from {username='slave'} at 127.0.0.1:34696
@ 0x7fe036878041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250901 19:54:34.457782 4902 ts_tablet_manager.cc:1918] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 19:54:34.457939 4902 log.cc:1199] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/wal/wals/a5067dd498ef4331bd9d9f3bcbcaec44
I20250901 19:54:34.457968 4811 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38242:
name: "default.Table_14959779c0504cb4bf46faa666221361"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:34.458410 4902 ts_tablet_manager.cc:1939] T a5067dd498ef4331bd9d9f3bcbcaec44 P c5b48d93671f4eb8980759b876e9ca21: Deleting consensus metadata
I20250901 19:54:34.458431 4808 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38242:
name: "default.Table_240cfd071a7c475bb44ff628d903bf9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:34.458762 4807 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38242:
name: "default.Table_b7d788fb008945558d1e6fd4dde4f706"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:34.459445 4902 tablet_replica.cc:331] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21: stopping tablet replica
I20250901 19:54:34.459352 4809 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38242:
table { table_name: "default.Table_0a1ca85b493d4aa285d85b17df52c4aa" } new_table_name: "default.Table_ed1ee4c8517b4614aafb4f78827e4f6e" modify_external_catalogs: true
I20250901 19:54:34.459671 4902 raft_consensus.cc:2241] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:34.459846 4902 raft_consensus.cc:2270] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:34.460456 4902 ts_tablet_manager.cc:1905] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:34.460680 4795 catalog_manager.cc:4928] TS c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): tablet a5067dd498ef4331bd9d9f3bcbcaec44 (table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]) successfully deleted
I20250901 19:54:34.462126 4904 tablet_replica.cc:331] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8: stopping tablet replica
I20250901 19:54:34.462348 4904 raft_consensus.cc:2241] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:34.462586 4904 raft_consensus.cc:2270] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:34.463215 4904 ts_tablet_manager.cc:1905] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fe03687fcbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250901 19:54:34.465611 4903 tablet_replica.cc:331] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734: stopping tablet replica
I20250901 19:54:34.465842 4903 raft_consensus.cc:2241] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [term 1 LEADER]: Raft consensus shutting down.
I20250901 19:54:34.466435 4903 raft_consensus.cc:2270] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:34.467085 4903 ts_tablet_manager.cc:1905] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fe03687fb39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250901 19:54:34.474812 4903 ts_tablet_manager.cc:1918] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 19:54:34.474958 4903 log.cc:1199] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/wal/wals/a5067dd498ef4331bd9d9f3bcbcaec44
I20250901 19:54:34.475472 4903 ts_tablet_manager.cc:1939] T a5067dd498ef4331bd9d9f3bcbcaec44 P d397574faadb441192c99d530f14a734: Deleting consensus metadata
I20250901 19:54:34.476347 3895 tablet_service.cc:1468] Processing CreateTablet for tablet 957e86dd3dd940fabcab40043e801e6b (DEFAULT_TABLE table=default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:34.476564 4903 tablet_replica.cc:331] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734: stopping tablet replica
I20250901 19:54:34.476783 4903 raft_consensus.cc:2241] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:34.476789 3895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 957e86dd3dd940fabcab40043e801e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:34.476847 3616 tablet_service.cc:1468] Processing CreateTablet for tablet 957e86dd3dd940fabcab40043e801e6b (DEFAULT_TABLE table=default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:34.477035 4903 raft_consensus.cc:2270] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:34.477295 3616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 957e86dd3dd940fabcab40043e801e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:34.477687 4903 ts_tablet_manager.cc:1905] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:34.480780 4902 ts_tablet_manager.cc:1918] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 19:54:34.480743 3757 tablet_service.cc:1468] Processing CreateTablet for tablet 957e86dd3dd940fabcab40043e801e6b (DEFAULT_TABLE table=default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:34.480939 4902 log.cc:1199] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/wal/wals/1d21a96219ab4f5f96bc6d9976f990ff
I20250901 19:54:34.481156 3757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 957e86dd3dd940fabcab40043e801e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:34.481392 4902 ts_tablet_manager.cc:1939] T 1d21a96219ab4f5f96bc6d9976f990ff P c5b48d93671f4eb8980759b876e9ca21: Deleting consensus metadata
I20250901 19:54:34.482842 3617 tablet_service.cc:1468] Processing CreateTablet for tablet 6579c1fbcfc04dac9345a8157b352591 (DEFAULT_TABLE table=default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:34.483322 3617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6579c1fbcfc04dac9345a8157b352591. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:34.483897 4795 catalog_manager.cc:4928] TS c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): tablet 1d21a96219ab4f5f96bc6d9976f990ff (table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]) successfully deleted
I20250901 19:54:34.483947 3758 tablet_service.cc:1468] Processing CreateTablet for tablet 6579c1fbcfc04dac9345a8157b352591 (DEFAULT_TABLE table=default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:34.484236 4903 ts_tablet_manager.cc:1918] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 19:54:34.484360 3758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6579c1fbcfc04dac9345a8157b352591. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:34.484376 4903 log.cc:1199] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/wal/wals/1d21a96219ab4f5f96bc6d9976f990ff
I20250901 19:54:34.485257 4903 ts_tablet_manager.cc:1939] T 1d21a96219ab4f5f96bc6d9976f990ff P d397574faadb441192c99d530f14a734: Deleting consensus metadata
I20250901 19:54:34.486981 4624 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:54:34.487176 4624 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:54:34.488489 4797 catalog_manager.cc:4928] TS d397574faadb441192c99d530f14a734 (127.2.246.194:33557): tablet a5067dd498ef4331bd9d9f3bcbcaec44 (table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]) successfully deleted
I20250901 19:54:34.488780 4797 catalog_manager.cc:4928] TS d397574faadb441192c99d530f14a734 (127.2.246.194:33557): tablet 1d21a96219ab4f5f96bc6d9976f990ff (table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]) successfully deleted
I20250901 19:54:34.489820 3896 tablet_service.cc:1468] Processing CreateTablet for tablet 6579c1fbcfc04dac9345a8157b352591 (DEFAULT_TABLE table=default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:34.490231 3896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6579c1fbcfc04dac9345a8157b352591. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fe0231a1cc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe0265c36db start_thread at ??:0
@ 0x7fe01f2f571f clone at ??:0
I20250901 19:54:34.501390 4904 ts_tablet_manager.cc:1918] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 19:54:34.501569 4904 log.cc:1199] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/wal/wals/a5067dd498ef4331bd9d9f3bcbcaec44
I20250901 19:54:34.502100 4904 ts_tablet_manager.cc:1939] T a5067dd498ef4331bd9d9f3bcbcaec44 P e0064a1b18e5470b85bde2277757aac8: Deleting consensus metadata
I20250901 19:54:34.503480 4904 tablet_replica.cc:331] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8: stopping tablet replica
I20250901 19:54:34.503888 4904 raft_consensus.cc:2241] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [term 1 LEADER]: Raft consensus shutting down.
I20250901 19:54:34.504354 4904 raft_consensus.cc:2270] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:34.505121 4904 ts_tablet_manager.cc:1905] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:34.509053 4907 tablet_bootstrap.cc:492] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:34.511730 4910 tablet_bootstrap.cc:492] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:34.512153 4907 tablet_bootstrap.cc:654] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:34.513459 4910 tablet_bootstrap.cc:654] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:34.528265 4909 tablet_bootstrap.cc:492] T 957e86dd3dd940fabcab40043e801e6b P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:34.529289 4907 tablet_bootstrap.cc:492] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:34.529462 4907 ts_tablet_manager.cc:1397] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250901 19:54:34.530313 4907 raft_consensus.cc:357] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.530481 4909 tablet_bootstrap.cc:654] T 957e86dd3dd940fabcab40043e801e6b P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:34.530527 4907 raft_consensus.cc:383] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:34.530611 4907 raft_consensus.cc:738] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:34.530871 4907 consensus_queue.cc:260] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.531894 4907 ts_tablet_manager.cc:1428] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250901 19:54:34.532397 4907 tablet_bootstrap.cc:492] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:34.532820 4904 ts_tablet_manager.cc:1918] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 19:54:34.532972 4904 log.cc:1199] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-2/wal/wals/1d21a96219ab4f5f96bc6d9976f990ff
I20250901 19:54:34.533433 4904 ts_tablet_manager.cc:1939] T 1d21a96219ab4f5f96bc6d9976f990ff P e0064a1b18e5470b85bde2277757aac8: Deleting consensus metadata
I20250901 19:54:34.534230 4907 tablet_bootstrap.cc:654] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:34.539348 4910 tablet_bootstrap.cc:492] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:34.539512 4910 ts_tablet_manager.cc:1397] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.012s
I20250901 19:54:34.540380 4910 raft_consensus.cc:357] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.540607 4910 raft_consensus.cc:383] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:34.540688 4907 tablet_bootstrap.cc:492] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:34.540697 4910 raft_consensus.cc:738] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:34.540918 4907 ts_tablet_manager.cc:1397] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250901 19:54:34.541009 4910 consensus_queue.cc:260] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.541864 4907 raft_consensus.cc:357] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.542079 4907 raft_consensus.cc:383] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:34.542181 4907 raft_consensus.cc:738] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:34.542438 4907 consensus_queue.cc:260] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.542797 4909 tablet_bootstrap.cc:492] T 957e86dd3dd940fabcab40043e801e6b P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:34.542941 4909 ts_tablet_manager.cc:1397] T 957e86dd3dd940fabcab40043e801e6b P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20250901 19:54:34.543099 4907 ts_tablet_manager.cc:1428] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250901 19:54:34.543761 4909 raft_consensus.cc:357] T 957e86dd3dd940fabcab40043e801e6b P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.543964 4909 raft_consensus.cc:383] T 957e86dd3dd940fabcab40043e801e6b P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:34.544054 4909 raft_consensus.cc:738] T 957e86dd3dd940fabcab40043e801e6b P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:34.544304 4909 consensus_queue.cc:260] T 957e86dd3dd940fabcab40043e801e6b P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.547036 4910 ts_tablet_manager.cc:1428] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20250901 19:54:34.547350 4910 tablet_bootstrap.cc:492] T 957e86dd3dd940fabcab40043e801e6b P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:34.548087 4909 ts_tablet_manager.cc:1428] T 957e86dd3dd940fabcab40043e801e6b P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 19:54:34.549175 4910 tablet_bootstrap.cc:654] T 957e86dd3dd940fabcab40043e801e6b P e0064a1b18e5470b85bde2277757aac8: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:34.549306 4909 tablet_bootstrap.cc:492] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:34.551146 4909 tablet_bootstrap.cc:654] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:34.559414 4909 tablet_bootstrap.cc:492] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:34.559692 4909 ts_tablet_manager.cc:1397] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20250901 19:54:34.560544 4909 raft_consensus.cc:357] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.560830 4909 raft_consensus.cc:383] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:34.560912 4909 raft_consensus.cc:738] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:34.561219 4909 consensus_queue.cc:260] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.561791 4909 ts_tablet_manager.cc:1428] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 19:54:34.563369 4910 tablet_bootstrap.cc:492] T 957e86dd3dd940fabcab40043e801e6b P e0064a1b18e5470b85bde2277757aac8: No bootstrap required, opened a new log
I20250901 19:54:34.564877 4910 ts_tablet_manager.cc:1397] T 957e86dd3dd940fabcab40043e801e6b P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.011s
I20250901 19:54:34.566040 4910 raft_consensus.cc:357] T 957e86dd3dd940fabcab40043e801e6b P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.566263 4910 raft_consensus.cc:383] T 957e86dd3dd940fabcab40043e801e6b P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:34.566345 4910 raft_consensus.cc:738] T 957e86dd3dd940fabcab40043e801e6b P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:34.566653 4910 consensus_queue.cc:260] T 957e86dd3dd940fabcab40043e801e6b P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.567206 4910 ts_tablet_manager.cc:1428] T 957e86dd3dd940fabcab40043e801e6b P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250901 19:54:34.573311 4607 raft_consensus.cc:491] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:34.573495 4607 raft_consensus.cc:513] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.574492 4607 leader_election.cc:290] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:34.575012 3637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6579c1fbcfc04dac9345a8157b352591" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21" is_pre_election: true
I20250901 19:54:34.575243 3637 raft_consensus.cc:2466] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 0.
I20250901 19:54:34.575417 3777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6579c1fbcfc04dac9345a8157b352591" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734" is_pre_election: true
I20250901 19:54:34.575672 3777 raft_consensus.cc:2466] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 0.
I20250901 19:54:34.576169 3849 leader_election.cc:304] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734, e0064a1b18e5470b85bde2277757aac8; no voters:
I20250901 19:54:34.576475 4607 raft_consensus.cc:2802] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:34.576588 4607 raft_consensus.cc:491] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:34.576689 4607 raft_consensus.cc:3058] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:34.579550 4607 raft_consensus.cc:513] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.580816 3777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6579c1fbcfc04dac9345a8157b352591" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734"
I20250901 19:54:34.581024 3777 raft_consensus.cc:3058] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:34.583674 3637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6579c1fbcfc04dac9345a8157b352591" candidate_uuid: "e0064a1b18e5470b85bde2277757aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21"
I20250901 19:54:34.583899 3637 raft_consensus.cc:3058] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:34.584030 3777 raft_consensus.cc:2466] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 1.
I20250901 19:54:34.584589 3851 leader_election.cc:304] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d397574faadb441192c99d530f14a734, e0064a1b18e5470b85bde2277757aac8; no voters:
I20250901 19:54:34.584802 4607 leader_election.cc:290] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [CANDIDATE]: Term 1 election: Requested vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:34.585139 4607 raft_consensus.cc:2802] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:34.585275 4607 raft_consensus.cc:695] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [term 1 LEADER]: Becoming Leader. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Running, Role: LEADER
I20250901 19:54:34.585603 4607 consensus_queue.cc:237] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.586678 3637 raft_consensus.cc:2466] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0064a1b18e5470b85bde2277757aac8 in term 1.
I20250901 19:54:34.718518 4100 raft_consensus.cc:491] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:34.718747 4100 raft_consensus.cc:513] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.719698 4100 leader_election.cc:290] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271), d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
W20250901 19:54:34.719962 3573 connection.cc:537] server connection from 127.0.0.1:32952 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:34.720203 3573 connection.cc:537] client connection to 127.2.246.253:34127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 19:54:34.720345 3914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "957e86dd3dd940fabcab40043e801e6b" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8" is_pre_election: true
I20250901 19:54:34.720567 3914 raft_consensus.cc:2466] T 957e86dd3dd940fabcab40043e801e6b P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
W20250901 19:54:34.720662 3712 connection.cc:537] server connection from 127.0.0.1:40758 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:34.720979 3973 connection.cc:537] client connection to 127.2.246.253:34127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:34.721117 3713 connection.cc:537] client connection to 127.2.246.253:34127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:34.721698 3851 connection.cc:537] client connection to 127.2.246.253:34127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 19:54:34.721889 3572 leader_election.cc:304] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, e0064a1b18e5470b85bde2277757aac8; no voters:
W20250901 19:54:34.721958 3851 connection.cc:537] server connection from 127.0.0.1:34696 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:34.722081 3973 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.246.253:34127)
I20250901 19:54:34.722415 4100 raft_consensus.cc:2802] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:34.722532 4100 raft_consensus.cc:491] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:34.722630 4100 raft_consensus.cc:3058] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
W20250901 19:54:34.722759 3973 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.246.253:34127)
W20250901 19:54:34.723152 3973 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.246.253:34127)
W20250901 19:54:34.723694 3973 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.246.253:34127)
I20250901 19:54:34.722265 3777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "957e86dd3dd940fabcab40043e801e6b" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734" is_pre_election: true
I20250901 19:54:34.723877 3777 raft_consensus.cc:2466] T 957e86dd3dd940fabcab40043e801e6b P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
I20250901 19:54:34.725546 4100 raft_consensus.cc:513] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.726150 4100 leader_election.cc:290] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Requested vote from peers e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271), d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:34.726657 3914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "957e86dd3dd940fabcab40043e801e6b" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0064a1b18e5470b85bde2277757aac8"
I20250901 19:54:34.726681 3777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "957e86dd3dd940fabcab40043e801e6b" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734"
I20250901 19:54:34.726850 3914 raft_consensus.cc:3058] T 957e86dd3dd940fabcab40043e801e6b P e0064a1b18e5470b85bde2277757aac8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:34.726855 3777 raft_consensus.cc:3058] T 957e86dd3dd940fabcab40043e801e6b P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:34.728776 3777 raft_consensus.cc:2466] T 957e86dd3dd940fabcab40043e801e6b P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
I20250901 19:54:34.728803 3914 raft_consensus.cc:2466] T 957e86dd3dd940fabcab40043e801e6b P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
I20250901 19:54:34.729197 3573 leader_election.cc:304] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters:
I20250901 19:54:34.729511 4100 raft_consensus.cc:2802] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:34.730520 4100 raft_consensus.cc:695] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [term 1 LEADER]: Becoming Leader. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Running, Role: LEADER
I20250901 19:54:34.730890 4100 consensus_queue.cc:237] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } }
I20250901 19:54:34.974112 4930 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:34.974370 4930 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:34.975121 4930 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:34.977516 4643 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:34.977775 4643 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:34.977890 4643 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:34.978157 4930 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250901 19:54:35.597083 4607 consensus_queue.cc:1035] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:35.604473 4939 consensus_queue.cc:1035] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:35.651575 3035 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu pid 4624
I20250901 19:54:35.662663 4687 generic_service.cc:196] Checking for leaks (request via RPC)
I20250901 19:54:35.772176 3035 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu with pid 4624
I20250901 19:54:35.772115 4948 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:35.789041 4100 consensus_queue.cc:1035] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:35.796694 4949 consensus_queue.cc:1035] T 957e86dd3dd940fabcab40043e801e6b P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 19:54:35.797241 3712 connection.cc:537] client connection to 127.2.246.252:37003 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:35.797641 3572 connection.cc:537] client connection to 127.2.246.252:37003 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:35.799176 4944 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 19:54:35.798156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.246.252:37003 (local address 127.2.246.194:37389)
0901 19:54:35.798343 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0901 19:54:35.798362 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0901 19:54:35.798481 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 19:54:35.798955 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":99}
I20250901 19:54:36.490531 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.252:37003
--webserver_interface=127.2.246.252
--webserver_port=36357
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250901 19:54:36.838622 4961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:36.838992 4961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:36.839118 4961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 19:54:36.849124 4961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 19:54:36.849236 4961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:36.849300 4961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 19:54:36.849367 4961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 19:54:36.860649 4961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.252:37003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.246.252
--webserver_port=36357
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:36.862406 4961 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:36.864478 4961 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 19:54:36.873800 4972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:36.873831 4973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:36.874207 4975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:36.875499 4961 server_base.cc:1047] running on GCE node
I20250901 19:54:36.876400 4961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:36.878111 4961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:36.879359 4961 hybrid_clock.cc:648] HybridClock initialized: now 1756756476879305 us; error 53 us; skew 500 ppm
I20250901 19:54:36.879936 4961 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:36.882515 4961 webserver.cc:480] Webserver started at http://127.2.246.252:36357/ using document root <none> and password file <none>
I20250901 19:54:36.883358 4961 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:36.883476 4961 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:36.889559 4961 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20250901 19:54:36.892938 4981 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:36.894532 4961 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250901 19:54:36.894726 4961 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
uuid: "08076b584d2a4aec8fd9868f85c9c7cd"
format_stamp: "Formatted at 2025-09-01 19:54:14 on dist-test-slave-jnn4"
I20250901 19:54:36.895473 4961 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:36.916090 4961 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:36.917021 4961 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:36.917321 4961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:36.947692 4961 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.252:37003
I20250901 19:54:36.947721 5034 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.252:37003 every 8 connection(s)
I20250901 19:54:36.949486 4961 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
I20250901 19:54:36.952167 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 4961
I20250901 19:54:36.957226 5035 sys_catalog.cc:263] Verifying existing consensus state
I20250901 19:54:36.960855 5035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Bootstrap starting.
I20250901 19:54:36.974260 5035 log.cc:826] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:37.017442 5035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=53 overwritten=0 applied=53 ignored=0} inserts{seen=30 ignored=0} mutations{seen=37 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 19:54:37.018098 5035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Bootstrap complete.
I20250901 19:54:37.024255 5035 raft_consensus.cc:357] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:37.025003 5035 raft_consensus.cc:738] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08076b584d2a4aec8fd9868f85c9c7cd, State: Initialized, Role: FOLLOWER
I20250901 19:54:37.025705 5035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 53, Last appended: 7.53, Last appended by leader: 53, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:37.026513 5039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:37.026770 5039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:37.027619 5035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: configured and running, proceeding with master startup.
I20250901 19:54:37.042073 5053 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:37.042273 5053 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: loading cluster ID for follower catalog manager: success
I20250901 19:54:37.047461 5053 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: acquiring CA information for follower catalog manager: success
I20250901 19:54:37.051177 5053 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 19:54:37.054654 4961 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:54:37.060058 4961 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:54:37.309791 3823 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
I20250901 19:54:37.313740 5000 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d397574faadb441192c99d530f14a734" instance_seqno: 1756756456460392) as {username='slave'} at 127.2.246.194:55443; Asking this server to re-register.
I20250901 19:54:37.314723 3823 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:37.317023 5000 ts_manager.cc:194] Registered new tserver with Master: d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:37.355883 3683 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
I20250901 19:54:37.357864 5000 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" instance_seqno: 1756756455802456) as {username='slave'} at 127.2.246.193:53193; Asking this server to re-register.
I20250901 19:54:37.358569 3683 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:37.360394 5000 ts_manager.cc:194] Registered new tserver with Master: c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:37.422916 3961 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
I20250901 19:54:37.424624 5000 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" instance_seqno: 1756756457020957) as {username='slave'} at 127.2.246.195:43417; Asking this server to re-register.
I20250901 19:54:37.425225 3961 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:37.426559 5000 ts_manager.cc:194] Registered new tserver with Master: e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:37.520807 5039 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:37.521117 5039 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:37.523188 5039 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:37.524055 4984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:37.525620 4984 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:37.525880 4984 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:37.526005 4984 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:37.526353 5039 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250901 19:54:38.264647 5066 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:38.264859 5066 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:38.265584 5066 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:38.268766 4984 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:38.269003 4984 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:38.269121 4984 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:38.269407 5066 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250901 19:54:39.131889 5082 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:39.132120 5082 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:39.132820 5082 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:39.135778 4984 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:54:39.136013 4984 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
I20250901 19:54:39.136107 4984 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:39.136422 5082 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250901 19:54:40.234236 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.253:34127
--webserver_interface=127.2.246.253
--webserver_port=36885
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250901 19:54:40.491607 5103 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:40.491961 5103 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:40.492710 5103 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:40.495065 4984 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:40.495342 4984 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:40.495460 4984 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:40.495725 5103 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250901 19:54:40.592727 5097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:40.593075 5097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:40.593158 5097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 19:54:40.602206 5097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 19:54:40.602332 5097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:40.602389 5097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 19:54:40.602437 5097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 19:54:40.613261 5097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.253:34127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.246.253
--webserver_port=36885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:40.614969 5097 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:40.616906 5097 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 19:54:40.625735 5110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:40.625768 5111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:40.626796 5113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:40.627238 5097 server_base.cc:1047] running on GCE node
I20250901 19:54:40.628124 5097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:40.629729 5097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:40.630985 5097 hybrid_clock.cc:648] HybridClock initialized: now 1756756480630928 us; error 38 us; skew 500 ppm
I20250901 19:54:40.631482 5097 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:40.633925 5097 webserver.cc:480] Webserver started at http://127.2.246.253:36885/ using document root <none> and password file <none>
I20250901 19:54:40.634729 5097 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:40.634827 5097 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:40.641294 5097 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20250901 19:54:40.644702 5119 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:40.646451 5097 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250901 19:54:40.646636 5097 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
uuid: "1d944ed54f8c43c9b8c4e7d589dad395"
format_stamp: "Formatted at 2025-09-01 19:54:14 on dist-test-slave-jnn4"
I20250901 19:54:40.647419 5097 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:40.678264 5097 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:40.679349 5097 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:40.679683 5097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:40.701197 5097 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.253:34127
I20250901 19:54:40.701226 5170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.253:34127 every 8 connection(s)
I20250901 19:54:40.703997 5097 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-1/data/info.pb
I20250901 19:54:40.706337 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 5097
I20250901 19:54:40.713632 5172 sys_catalog.cc:263] Verifying existing consensus state
I20250901 19:54:40.717847 5172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Bootstrap starting.
I20250901 19:54:40.732376 5172 log.cc:826] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:40.775560 5172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Bootstrap replayed 1/1 log segments. Stats: ops{read=55 overwritten=2 applied=53 ignored=0} inserts{seen=30 ignored=0} mutations{seen=37 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 19:54:40.776312 5172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Bootstrap complete.
I20250901 19:54:40.786239 5172 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:40.787429 5172 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d944ed54f8c43c9b8c4e7d589dad395, State: Initialized, Role: FOLLOWER
I20250901 19:54:40.788339 5172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 53, Last appended: 7.53, Last appended by leader: 53, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:40.789572 5177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:40.789990 5177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:40.790808 5172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 19:54:40.809494 5191 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:40.809767 5191 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: loading cluster ID for follower catalog manager: success
I20250901 19:54:40.803931 5097 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:54:40.811040 5097 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:54:40.833976 5191 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: acquiring CA information for follower catalog manager: success
I20250901 19:54:40.844599 5191 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 19:54:40.847715 3829 heartbeater.cc:344] Connected to a master server at 127.2.246.253:34127
I20250901 19:54:40.851619 5136 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d397574faadb441192c99d530f14a734" instance_seqno: 1756756456460392) as {username='slave'} at 127.2.246.194:58921; Asking this server to re-register.
I20250901 19:54:40.852941 3829 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:40.855401 5136 ts_manager.cc:194] Registered new tserver with Master: d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:40.871202 3685 heartbeater.cc:344] Connected to a master server at 127.2.246.253:34127
I20250901 19:54:40.873706 5136 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" instance_seqno: 1756756455802456) as {username='slave'} at 127.2.246.193:55541; Asking this server to re-register.
I20250901 19:54:40.874485 3685 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:40.876324 5136 ts_manager.cc:194] Registered new tserver with Master: c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:40.941537 3963 heartbeater.cc:344] Connected to a master server at 127.2.246.253:34127
I20250901 19:54:40.943217 5136 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" instance_seqno: 1756756457020957) as {username='slave'} at 127.2.246.195:39767; Asking this server to re-register.
I20250901 19:54:40.943971 3963 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:40.945526 5136 ts_manager.cc:194] Registered new tserver with Master: e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:41.136093 5201 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:41.136287 5201 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:41.137024 5201 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:41.142891 4984 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
I20250901 19:54:41.148361 5146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 53 } } ignore_live_leader: false dest_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" is_pre_election: true
I20250901 19:54:41.149106 5146 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08076b584d2a4aec8fd9868f85c9c7cd in term 7.
I20250901 19:54:41.149936 4984 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 4c9163569ae1418090a245256e263186
I20250901 19:54:41.150315 5201 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250901 19:54:41.150451 5201 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:41.150561 5201 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 7 FOLLOWER]: Advancing to term 8
I20250901 19:54:41.154505 5201 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:41.155292 5201 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 election: Requested vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
I20250901 19:54:41.156417 5146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 53 } } ignore_live_leader: false dest_uuid: "1d944ed54f8c43c9b8c4e7d589dad395"
I20250901 19:54:41.156688 5146 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 7 FOLLOWER]: Advancing to term 8
I20250901 19:54:41.160909 5146 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08076b584d2a4aec8fd9868f85c9c7cd in term 8.
I20250901 19:54:41.161424 4984 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395; no voters:
W20250901 19:54:41.161744 4984 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
I20250901 19:54:41.161774 5201 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 8 FOLLOWER]: Leader election won for term 8
I20250901 19:54:41.162312 5201 raft_consensus.cc:695] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 8 LEADER]: Becoming Leader. State: Replica: 08076b584d2a4aec8fd9868f85c9c7cd, State: Running, Role: LEADER
I20250901 19:54:41.162961 5201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 7.53, Last appended by leader: 53, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:41.166462 5205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08076b584d2a4aec8fd9868f85c9c7cd. Latest consensus state: current_term: 8 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:41.166823 5205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:41.167560 5207 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 19:54:41.170320 5207 catalog_manager.cc:671] Loaded metadata for table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077]
I20250901 19:54:41.170837 5207 catalog_manager.cc:671] Loaded metadata for table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]
I20250901 19:54:41.171422 5207 catalog_manager.cc:671] Loaded metadata for table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]
I20250901 19:54:41.171973 5207 catalog_manager.cc:671] Loaded metadata for table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67]
I20250901 19:54:41.172428 5207 catalog_manager.cc:671] Loaded metadata for table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]
I20250901 19:54:41.172917 5207 catalog_manager.cc:671] Loaded metadata for table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]
I20250901 19:54:41.173339 5207 catalog_manager.cc:671] Loaded metadata for table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8]
I20250901 19:54:41.173784 5207 catalog_manager.cc:671] Loaded metadata for table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616]
I20250901 19:54:41.174154 5207 catalog_manager.cc:671] Loaded metadata for table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]
I20250901 19:54:41.177392 5207 tablet_loader.cc:96] loaded metadata for tablet 5d8e6708fad646d39f3cd20381c1c8aa (table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67])
I20250901 19:54:41.178089 5207 tablet_loader.cc:96] loaded metadata for tablet 6579c1fbcfc04dac9345a8157b352591 (table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612])
I20250901 19:54:41.178453 5207 tablet_loader.cc:96] loaded metadata for tablet 7ab5ab06eece48fdb346f232247f72e0 (table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077])
I20250901 19:54:41.179018 5207 tablet_loader.cc:96] loaded metadata for tablet 957e86dd3dd940fabcab40043e801e6b (table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896])
I20250901 19:54:41.179512 5207 tablet_loader.cc:96] loaded metadata for tablet b867925c426149c99ffdd6d5df396462 (table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6])
I20250901 19:54:41.180055 5207 tablet_loader.cc:96] loaded metadata for tablet e003ca1913284f239f5b0474fc621abe (table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8])
I20250901 19:54:41.180457 5207 tablet_loader.cc:96] loaded metadata for tablet e0063b69ff9a48f6a1229cd26704c66d (table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59])
I20250901 19:54:41.180768 5207 tablet_loader.cc:96] loaded metadata for tablet e1d516f8388c4a099f81d6e5a3c7cf4d (table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3])
I20250901 19:54:41.181420 5207 tablet_loader.cc:96] loaded metadata for tablet f4fe9cadf6414b548cbd95572d9ed1c8 (table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616])
I20250901 19:54:41.182214 5207 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 19:54:41.183741 5207 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:41.183821 5207 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 19:54:41.185233 5207 catalog_manager.cc:1506] Loading token signing keys...
I20250901 19:54:41.186259 5207 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Loaded TSK: 0
I20250901 19:54:41.187206 5207 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 19:54:41.188540 5207 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 19:54:41.189503 5207 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250901 19:54:41.301124 4999 master_service.cc:948] ReplaceTablet: received request to replace tablet f4fe9cadf6414b548cbd95572d9ed1c8 from {username='slave'} at 127.0.0.1:33016
I20250901 19:54:41.301250 5000 master_service.cc:948] ReplaceTablet: received request to replace tablet 7b163f22e81940f2a8d9249c5cb64afc from {username='slave'} at 127.0.0.1:33016
I20250901 19:54:41.301589 4997 master_service.cc:948] ReplaceTablet: received request to replace tablet e1d516f8388c4a099f81d6e5a3c7cf4d from {username='slave'} at 127.0.0.1:33016
I20250901 19:54:41.301831 4996 master_service.cc:948] ReplaceTablet: received request to replace tablet f4fe9cadf6414b548cbd95572d9ed1c8 from {username='slave'} at 127.0.0.1:33016
I20250901 19:54:41.302222 4998 master_service.cc:948] ReplaceTablet: received request to replace tablet f4fe9cadf6414b548cbd95572d9ed1c8 from {username='slave'} at 127.0.0.1:33016
I20250901 19:54:41.302922 5000 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33016:
name: "default.Table_edff33351bd64d16aec20db4230ac013"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:41.302435 4995 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33016:
name: "default.Table_8728509cfe374d5ea49110ea201c1659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:41.306216 4991 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33016:
table { table_name: "default.Table_079afb395d5c4c17a18fd7e77548109d" } modify_external_catalogs: true
I20250901 19:54:41.305410 4994 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33016:
name: "default.Table_69070232fcfc4853b5fff18e23fbf965"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:41.306248 4992 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33016:
table { table_name: "default.Table_3cbad8a0189443d9b60b0abc4460862a" } modify_external_catalogs: true
I20250901 19:54:41.306978 4991 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33016:
table { table_name: "default.Table_079afb395d5c4c17a18fd7e77548109d" } modify_external_catalogs: true
I20250901 19:54:41.312559 4992 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33016:
table { table_name: "default.Table_3cbad8a0189443d9b60b0abc4460862a" } modify_external_catalogs: true
I20250901 19:54:41.304484 4993 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33016:
name: "default.Table_788e14a6980547248f5627f671a238a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250901 19:54:41.313823 5052 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_0a1ca85b493d4aa285d85b17df52c4aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_0a1ca85b493d4aa285d85b17df52c4aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0a1ca85b493d4aa285d85b17df52c4aa\",\"kudu.table_id\":\"fe4c5d5ccf384fbfbf05cb7b4f0a1bb6\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_c32254b687594ed89daf3edaded35200, message={"server":"","servicePrincipal":"","db":"default","table":"table_c32254b687594ed89daf3edaded35200","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c32254b687594ed89daf3edaded35200\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_c32254b687594ed89daf3edaded35200\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c32254b687594ed89daf3edaded35200\",\"kudu.table_id\":\"268741b95ddb403f8f59e4219723950e\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f2380233dcd google::LogMessage::Fail() at ??:0
@ 0x7f2380237b93 google::LogMessage::SendToLog() at ??:0
I20250901 19:54:41.319446 5146 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Refusing update from remote peer 08076b584d2a4aec8fd9868f85c9c7cd: Log matching property violated. Preceding OpId in replica: term: 7 index: 53. Preceding OpId from leader: term: 8 index: 55. (index mismatch)
@ 0x7f23802337cc google::LogMessage::Flush() at ??:0
I20250901 19:54:41.320936 5210 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
@ 0x7f2380233a11 google::LogMessage::~LogMessage() at ??:0
W20250901 19:54:41.321866 4984 consensus_peers.cc:489] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd -> Peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Couldn't send request to peer 4c9163569ae1418090a245256e263186. Status: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f2396668794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f2396667041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250901 19:54:41.335122 5211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08076b584d2a4aec8fd9868f85c9c7cd. Latest consensus state: current_term: 8 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:41.335419 5211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f239666ecbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f239666eb39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250901 19:54:41.353034 4997 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Sending DeleteTablet for 3 replicas of tablet e1d516f8388c4a099f81d6e5a3c7cf4d
I20250901 19:54:41.356222 4997 catalog_manager.cc:6652] ReplaceTablet: tablet e1d516f8388c4a099f81d6e5a3c7cf4d deleted and replaced by tablet 59eb68e59e59483fa9c3999da17aeaec
I20250901 19:54:41.359031 4999 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Sending DeleteTablet for 3 replicas of tablet f4fe9cadf6414b548cbd95572d9ed1c8
@ 0x7f2382f90cc9 kudu::Thread::SuperviseThread() at ??:0
I20250901 19:54:41.361589 5211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:41.361847 5211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:41.363006 4999 catalog_manager.cc:6652] ReplaceTablet: tablet f4fe9cadf6414b548cbd95572d9ed1c8 deleted and replaced by tablet 8b9fa477370948adb7af0b9d29734522
@ 0x7f23863b26db start_thread at ??:0
@ 0x7f237f0e471f clone at ??:0
W20250901 19:54:41.549029 3850 connection.cc:537] client connection to 127.2.246.252:37003 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:41.549393 5225 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 19:54:41.356831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.2.246.195:33271)
0901 19:54:41.372295 (+ 15464us) server_negotiation.cc:206] Beginning negotiation
0901 19:54:41.372304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0901 19:54:41.549129 (+176825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43434: BlockingRecv error: recv got EOF from 127.0.0.1:43434 (error 108)
Metrics: {"server-negotiator.queue_time_us":15275,"thread_start_us":15072,"threads_started":1}
W20250901 19:54:41.549460 5224 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 19:54:41.356496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.2.246.194:33557)
0901 19:54:41.372456 (+ 15960us) server_negotiation.cc:206] Beginning negotiation
0901 19:54:41.372462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 19:54:41.372482 (+ 20us) server_negotiation.cc:407] Connection header received
0901 19:54:41.372558 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 19:54:41.372564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 19:54:41.372649 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 19:54:41.372752 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 19:54:41.549171 (+176419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36092: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":15854,"thread_start_us":15756,"threads_started":1}
W20250901 19:54:41.549504 5223 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 19:54:41.356105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34724 (local address 127.2.246.193:37767)
0901 19:54:41.369219 (+ 13114us) server_negotiation.cc:206] Beginning negotiation
0901 19:54:41.369223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 19:54:41.369241 (+ 18us) server_negotiation.cc:407] Connection header received
0901 19:54:41.369312 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 19:54:41.369316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 19:54:41.369378 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 19:54:41.369474 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 19:54:41.549128 (+179654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34724: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":13038,"thread_start_us":13520,"threads_started":1}
W20250901 19:54:41.550056 3972 connection.cc:537] client connection to 127.2.246.252:37003 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:41.550477 3572 connection.cc:537] client connection to 127.2.246.252:37003 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:41.550740 3972 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.246.252:37003)
W20250901 19:54:41.551055 3972 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.246.252:37003)
W20250901 19:54:41.551275 3972 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.246.252:37003)
W20250901 19:54:41.551738 3972 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.246.252:37003)
I20250901 19:54:41.839144 5211 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:41.839411 5211 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:41.841145 5211 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:41.842069 5122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:41.843550 5122 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:41.843788 5121 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:41.843930 5121 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186
I20250901 19:54:41.844367 5211 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250901 19:54:42.357473 5241 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:42.357659 5241 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:42.358354 5241 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:42.360502 5121 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
W20250901 19:54:42.360738 5122 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
I20250901 19:54:42.360874 5122 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186
I20250901 19:54:42.361137 5241 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250901 19:54:43.015424 5246 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:43.015619 5246 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:43.016371 5246 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:43.018682 5121 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
W20250901 19:54:43.018913 5122 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
I20250901 19:54:43.019037 5122 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186
I20250901 19:54:43.019331 5246 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250901 19:54:43.645417 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.254:33523
--webserver_interface=127.2.246.254
--webserver_port=40931
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250901 19:54:43.997238 5251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:43.997570 5251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:43.997710 5251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 19:54:44.006412 5251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 19:54:44.006518 5251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:44.006574 5251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 19:54:44.006644 5251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 19:54:44.016968 5251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.254:33523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.246.254
--webserver_port=40931
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:44.018657 5251 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:44.020529 5251 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 19:54:44.029356 5256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:44.029371 5257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:44.030309 5259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:44.031080 5251 server_base.cc:1047] running on GCE node
I20250901 19:54:44.031989 5251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:44.033545 5251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:44.034755 5251 hybrid_clock.cc:648] HybridClock initialized: now 1756756484034701 us; error 51 us; skew 500 ppm
I20250901 19:54:44.035145 5251 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:44.037487 5251 webserver.cc:480] Webserver started at http://127.2.246.254:40931/ using document root <none> and password file <none>
I20250901 19:54:44.038292 5251 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:44.038405 5251 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:44.044245 5251 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20250901 19:54:44.047386 5265 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:44.048979 5251 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250901 19:54:44.049165 5251 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
uuid: "4c9163569ae1418090a245256e263186"
format_stamp: "Formatted at 2025-09-01 19:54:13 on dist-test-slave-jnn4"
I20250901 19:54:44.049926 5251 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:44.071050 5251 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:44.072135 5251 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:44.072454 5251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:44.093538 5251 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.254:33523
I20250901 19:54:44.093570 5316 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.254:33523 every 8 connection(s)
I20250901 19:54:44.095388 5251 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-0/data/info.pb
I20250901 19:54:44.097602 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 5251
I20250901 19:54:44.104221 5317 sys_catalog.cc:263] Verifying existing consensus state
I20250901 19:54:44.108240 5320 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:44.108438 5320 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:44.109081 5317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Bootstrap starting.
I20250901 19:54:44.113150 5320 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:44.118012 5121 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:44.125641 5317 log.cc:826] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:44.162886 5317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Bootstrap replayed 1/1 log segments. Stats: ops{read=45 overwritten=0 applied=45 ignored=0} inserts{seen=26 ignored=0} mutations{seen=29 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 19:54:44.163517 5317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Bootstrap complete.
I20250901 19:54:44.168900 5317 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:44.169582 5317 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9163569ae1418090a245256e263186, State: Initialized, Role: FOLLOWER
I20250901 19:54:44.170234 5317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 5.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:44.171168 5326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:44.171552 5326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:44.172413 5317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 19:54:44.172468 5292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" candidate_term: 9 candidate_status { last_received { term: 8 index: 56 } } ignore_live_leader: false dest_uuid: "4c9163569ae1418090a245256e263186" is_pre_election: true
I20250901 19:54:44.173053 5292 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d944ed54f8c43c9b8c4e7d589dad395 in term 5.
I20250901 19:54:44.173766 5122 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186; no voters: 08076b584d2a4aec8fd9868f85c9c7cd
I20250901 19:54:44.174110 5320 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250901 19:54:44.174242 5320 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 08076b584d2a4aec8fd9868f85c9c7cd)
I20250901 19:54:44.174331 5320 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 8 FOLLOWER]: Advancing to term 9
I20250901 19:54:44.176508 5320 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:44.177618 5320 leader_election.cc:290] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 election: Requested vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
I20250901 19:54:44.177790 5292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" candidate_term: 9 candidate_status { last_received { term: 8 index: 56 } } ignore_live_leader: false dest_uuid: "4c9163569ae1418090a245256e263186"
I20250901 19:54:44.177991 5292 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 5 FOLLOWER]: Advancing to term 9
W20250901 19:54:44.178895 5121 leader_election.cc:336] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:44.182263 5292 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d944ed54f8c43c9b8c4e7d589dad395 in term 9.
I20250901 19:54:44.182832 5122 leader_election.cc:304] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186; no voters: 08076b584d2a4aec8fd9868f85c9c7cd
I20250901 19:54:44.183204 5320 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 9 FOLLOWER]: Leader election won for term 9
I20250901 19:54:44.183729 5320 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [term 9 LEADER]: Becoming Leader. State: Replica: 1d944ed54f8c43c9b8c4e7d589dad395, State: Running, Role: LEADER
I20250901 19:54:44.184288 5320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 8.56, Last appended by leader: 56, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:44.188505 5337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d944ed54f8c43c9b8c4e7d589dad395. Latest consensus state: current_term: 9 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:44.188823 5337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:44.192028 5344 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 19:54:44.198350 5344 catalog_manager.cc:671] Loaded metadata for table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077]
I20250901 19:54:44.199039 5344 catalog_manager.cc:671] Loaded metadata for table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]
I20250901 19:54:44.199201 5341 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:44.199291 5341 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: loading cluster ID for follower catalog manager: success
I20250901 19:54:44.199472 5344 catalog_manager.cc:671] Loaded metadata for table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]
I20250901 19:54:44.199859 5344 catalog_manager.cc:671] Loaded metadata for table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67]
I20250901 19:54:44.200148 5344 catalog_manager.cc:671] Loaded metadata for table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]
I20250901 19:54:44.200574 5344 catalog_manager.cc:671] Loaded metadata for table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]
I20250901 19:54:44.201022 5344 catalog_manager.cc:671] Loaded metadata for table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8]
I20250901 19:54:44.201407 5344 catalog_manager.cc:671] Loaded metadata for table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616]
I20250901 19:54:44.201889 5344 catalog_manager.cc:671] Loaded metadata for table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]
I20250901 19:54:44.204864 5344 tablet_loader.cc:96] loaded metadata for tablet 59eb68e59e59483fa9c3999da17aeaec (table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3])
I20250901 19:54:44.205287 5344 tablet_loader.cc:96] loaded metadata for tablet 5d8e6708fad646d39f3cd20381c1c8aa (table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67])
I20250901 19:54:44.205878 5344 tablet_loader.cc:96] loaded metadata for tablet 6579c1fbcfc04dac9345a8157b352591 (table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612])
I20250901 19:54:44.206264 5344 tablet_loader.cc:96] loaded metadata for tablet 7ab5ab06eece48fdb346f232247f72e0 (table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077])
I20250901 19:54:44.206816 5344 tablet_loader.cc:96] loaded metadata for tablet 8b9fa477370948adb7af0b9d29734522 (table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616])
I20250901 19:54:44.207263 5344 tablet_loader.cc:96] loaded metadata for tablet 957e86dd3dd940fabcab40043e801e6b (table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896])
I20250901 19:54:44.208137 5344 tablet_loader.cc:96] loaded metadata for tablet b867925c426149c99ffdd6d5df396462 (table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6])
I20250901 19:54:44.208532 5344 tablet_loader.cc:96] loaded metadata for tablet e003ca1913284f239f5b0474fc621abe (table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8])
I20250901 19:54:44.208961 5344 tablet_loader.cc:96] loaded metadata for tablet e0063b69ff9a48f6a1229cd26704c66d (table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59])
I20250901 19:54:44.210317 5344 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 19:54:44.211895 5344 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:44.211992 5344 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 19:54:44.213936 5344 catalog_manager.cc:1506] Loading token signing keys...
I20250901 19:54:44.215231 5344 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Loaded TSK: 0
I20250901 19:54:44.216216 5341 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: acquiring CA information for follower catalog manager: success
I20250901 19:54:44.216467 5344 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 19:54:44.217572 5344 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 19:54:44.219115 5344 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250901 19:54:44.219869 5341 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 19:54:44.228976 5251 master_runner.cc:186] Error getting master registration for 127.2.246.252:37003: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111),
I20250901 19:54:44.233002 5251 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:54:44.288010 5136 master_service.cc:948] ReplaceTablet: received request to replace tablet 7ab5ab06eece48fdb346f232247f72e0 from {username='slave'} at 127.0.0.1:51546
I20250901 19:54:44.289351 5128 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51546:
table { table_name: "default.Table_ea3686ff3ba94b9a984657e263ca0832" } new_table_name: "default.Table_f79692d68cd04e74ae3e3c67bc4f99a3" modify_external_catalogs: true
I20250901 19:54:44.290171 5129 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51546:
table { table_name: "default.Table_d29d54fc138b40009682d5b2585d25bd" } new_table_name: "default.Table_6a818be47b2e41e48156df825fbe0cc3" modify_external_catalogs: true
I20250901 19:54:44.290841 5130 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51546:
table { table_name: "default.Table_5cf2893927ed42d597f83e0a04da2245" } new_table_name: "default.Table_3d911927cc9648b39bc8fae3c7d25c87" modify_external_catalogs: true
I20250901 19:54:44.293740 5127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51546:
name: "default.Table_8728509cfe374d5ea49110ea201c1659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:44.295099 5131 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51546:
table { table_name: "default.Table_0a1ca85b493d4aa285d85b17df52c4aa" } new_table_name: "default.Table_ed1ee4c8517b4614aafb4f78827e4f6e" modify_external_catalogs: true
I20250901 19:54:44.295454 5135 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51546:
table { table_name: "default.Table_51c52d3eec714bbd9a77bed453c02bb1" } modify_external_catalogs: true
I20250901 19:54:44.295702 5135 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51546:
table { table_name: "default.Table_51c52d3eec714bbd9a77bed453c02bb1" } modify_external_catalogs: true
I20250901 19:54:44.294595 5134 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51546:
name: "default.Table_240cfd071a7c475bb44ff628d903bf9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:44.295939 5132 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51546:
name: "default.Table_b7d788fb008945558d1e6fd4dde4f706"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:44.296787 5133 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51546:
name: "default.Table_14959779c0504cb4bf46faa666221361"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:44.301152 5292 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Refusing update from remote peer 1d944ed54f8c43c9b8c4e7d589dad395: Log matching property violated. Preceding OpId in replica: term: 5 index: 45. Preceding OpId from leader: term: 9 index: 58. (index mismatch)
W20250901 19:54:44.302107 5121 consensus_peers.cc:489] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 -> Peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Couldn't send request to peer 08076b584d2a4aec8fd9868f85c9c7cd. Status: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 19:54:44.302196 5337 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 45, Time since last communication: 0.000s
F20250901 19:54:44.302590 5190 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_0a1ca85b493d4aa285d85b17df52c4aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_0a1ca85b493d4aa285d85b17df52c4aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0a1ca85b493d4aa285d85b17df52c4aa\",\"kudu.table_id\":\"fe4c5d5ccf384fbfbf05cb7b4f0a1bb6\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_c32254b687594ed89daf3edaded35200, message={"server":"","servicePrincipal":"","db":"default","table":"table_c32254b687594ed89daf3edaded35200","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c32254b687594ed89daf3edaded35200\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_c32254b687594ed89daf3edaded35200\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c32254b687594ed89daf3edaded35200\",\"kudu.table_id\":\"268741b95ddb403f8f59e4219723950e\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f47932d9dcd google::LogMessage::Fail() at ??:0
@ 0x7f47932ddb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f47932d97cc google::LogMessage::Flush() at ??:0
@ 0x7f47932d9a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f47a970e794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f47a970d041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250901 19:54:44.324117 5350 mvcc.cc:204] Tried to move back new op lower bound from 7195674519402356736 to 7195674500367917056. Current Snapshot: MvccSnapshot[applied={T|T < 7195674519402356736 or (T in {7195674519402356736})}]
I20250901 19:54:44.327147 5355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:44.327584 5355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:44.328063 5353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:44.328394 5353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f47a9714cbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250901 19:54:44.328893 5326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d944ed54f8c43c9b8c4e7d589dad395. Latest consensus state: current_term: 9 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:44.329114 5326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:44.329387 5358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:44.329722 5358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f47a9714b39 std::_Function_handler<>::_M_invoke() at ??:0
I20250901 19:54:44.338049 5358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:44.338356 5358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: FOLLOWER
@ 0xa774dd std::function<>::operator()() at ??:?
I20250901 19:54:44.346798 5136 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395: Sending DeleteTablet for 3 replicas of tablet 7ab5ab06eece48fdb346f232247f72e0
@ 0x7f4796036cc9 kudu::Thread::SuperviseThread() at ??:0
I20250901 19:54:44.348143 5337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:44.348390 5337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d944ed54f8c43c9b8c4e7d589dad395 [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:44.349488 5136 catalog_manager.cc:6652] ReplaceTablet: tablet 7ab5ab06eece48fdb346f232247f72e0 deleted and replaced by tablet b75452cf2414469185c3a763d5a39cac
@ 0x7f47994586db start_thread at ??:0
@ 0x7f479218a71f clone at ??:0
I20250901 19:54:44.358003 3824 heartbeater.cc:344] Connected to a master server at 127.2.246.254:33523
I20250901 19:54:44.358253 3824 heartbeater.cc:499] Master 127.2.246.254:33523 was elected leader, sending a full tablet report...
I20250901 19:54:44.362769 5282 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d397574faadb441192c99d530f14a734" instance_seqno: 1756756456460392) as {username='slave'} at 127.2.246.194:56911; Asking this server to re-register.
I20250901 19:54:44.364015 3824 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:44.366412 5282 ts_manager.cc:194] Registered new tserver with Master: d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:44.384060 3684 heartbeater.cc:344] Connected to a master server at 127.2.246.254:33523
I20250901 19:54:44.386358 5282 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" instance_seqno: 1756756455802456) as {username='slave'} at 127.2.246.193:48615; Asking this server to re-register.
I20250901 19:54:44.387382 3684 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:44.389304 5282 ts_manager.cc:194] Registered new tserver with Master: c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:44.457906 3962 heartbeater.cc:344] Connected to a master server at 127.2.246.254:33523
I20250901 19:54:44.460204 5282 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" instance_seqno: 1756756457020957) as {username='slave'} at 127.2.246.195:55065; Asking this server to re-register.
I20250901 19:54:44.461035 3962 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:44.462765 5282 ts_manager.cc:194] Registered new tserver with Master: e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
W20250901 19:54:44.548982 5365 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 19:54:44.352726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34738 (local address 127.2.246.193:37767)
0901 19:54:44.361625 (+ 8899us) server_negotiation.cc:206] Beginning negotiation
0901 19:54:44.361633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0901 19:54:44.548649 (+187016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34738: BlockingRecv error: recv got EOF from 127.0.0.1:34738 (error 108)
Metrics: {"server-negotiator.queue_time_us":7378,"thread_start_us":7262,"threads_started":1}
W20250901 19:54:44.549021 5364 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 19:54:44.351397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.2.246.195:33271)
0901 19:54:44.356807 (+ 5410us) server_negotiation.cc:206] Beginning negotiation
0901 19:54:44.356813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 19:54:44.356833 (+ 20us) server_negotiation.cc:407] Connection header received
0901 19:54:44.356904 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 19:54:44.356911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 19:54:44.357020 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 19:54:44.357141 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 19:54:44.548650 (+191509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43438: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":5263,"thread_start_us":5174,"threads_started":1}
W20250901 19:54:44.549024 5366 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 19:54:44.353126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36104 (local address 127.2.246.194:33557)
0901 19:54:44.361349 (+ 8223us) server_negotiation.cc:206] Beginning negotiation
0901 19:54:44.361355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 19:54:44.361375 (+ 20us) server_negotiation.cc:407] Connection header received
0901 19:54:44.361457 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 19:54:44.361462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 19:54:44.361747 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 19:54:44.361861 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 19:54:44.548669 (+186808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36104: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":8095,"thread_start_us":8977,"threads_started":1}
W20250901 19:54:44.549453 3851 connection.cc:537] client connection to 127.2.246.253:34127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:44.549954 3713 connection.cc:537] client connection to 127.2.246.253:34127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:44.550590 3573 connection.cc:537] client connection to 127.2.246.253:34127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:44.550863 3973 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.246.253:34127)
W20250901 19:54:44.551471 3973 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.246.253:34127)
W20250901 19:54:44.551806 3973 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.246.253:34127)
W20250901 19:54:44.552718 3973 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.246.253:34127)
I20250901 19:54:44.832901 5358 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:44.833149 5358 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:44.835019 5358 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:44.836138 5267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
W20250901 19:54:44.836130 5268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:54:44.837832 5268 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:54:44.838044 5267 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:44.838210 5267 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c9163569ae1418090a245256e263186; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395
I20250901 19:54:44.838570 5358 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 19:54:45.564464 5380 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:45.564738 5380 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:45.565454 5380 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:45.567557 5268 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:54:45.567739 5267 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
I20250901 19:54:45.567940 5267 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c9163569ae1418090a245256e263186; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395
I20250901 19:54:45.568210 5380 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 19:54:46.100973 5387 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:46.101152 5387 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:46.101848 5387 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:46.104005 5267 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
W20250901 19:54:46.104194 5268 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:46.104362 5268 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c9163569ae1418090a245256e263186; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395
I20250901 19:54:46.104619 5387 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 19:54:46.631444 5391 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:46.631719 5391 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:46.632364 5391 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:46.634479 5267 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003): Network error: Client connection negotiation failed: client connection to 127.2.246.252:37003: connect: Connection refused (error 111)
W20250901 19:54:46.634788 5268 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:46.634967 5268 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c9163569ae1418090a245256e263186; no voters: 08076b584d2a4aec8fd9868f85c9c7cd, 1d944ed54f8c43c9b8c4e7d589dad395
I20250901 19:54:46.635258 5391 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250901 19:54:46.906359 3035 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
/tmp/dist-test-taskaPZSUq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.252:37003
--webserver_interface=127.2.246.252
--webserver_port=36357
--builtin_ntp_servers=127.2.246.212:40595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250901 19:54:47.281971 5395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 19:54:47.282320 5395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 19:54:47.282449 5395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 19:54:47.291061 5395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 19:54:47.291203 5395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 19:54:47.291275 5395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 19:54:47.291343 5395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 19:54:47.301579 5395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.246.212:40595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41647
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.246.252:37003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.246.252
--webserver_port=36357
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 19:43:24 UTC on 24a791456cd2
build id 7852
ASAN enabled
I20250901 19:54:47.303260 5395 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 19:54:47.305490 5395 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 19:54:47.313903 5402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:47.314189 5403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 19:54:47.315717 5405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 19:54:47.316058 5395 server_base.cc:1047] running on GCE node
I20250901 19:54:47.316897 5395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 19:54:47.318461 5395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 19:54:47.319743 5395 hybrid_clock.cc:648] HybridClock initialized: now 1756756487319689 us; error 47 us; skew 500 ppm
I20250901 19:54:47.320127 5395 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 19:54:47.322592 5395 webserver.cc:480] Webserver started at http://127.2.246.252:36357/ using document root <none> and password file <none>
I20250901 19:54:47.323387 5395 fs_manager.cc:362] Metadata directory not provided
I20250901 19:54:47.323508 5395 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 19:54:47.330102 5395 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250901 19:54:47.335734 5411 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 19:54:47.337397 5395 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250901 19:54:47.337612 5395 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data,/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
uuid: "08076b584d2a4aec8fd9868f85c9c7cd"
format_stamp: "Formatted at 2025-09-01 19:54:14 on dist-test-slave-jnn4"
I20250901 19:54:47.338388 5395 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 19:54:47.373066 5395 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 19:54:47.374195 5395 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 19:54:47.374725 5395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 19:54:47.397660 5395 rpc_server.cc:307] RPC server started. Bound to: 127.2.246.252:37003
I20250901 19:54:47.397660 5462 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.246.252:37003 every 8 connection(s)
I20250901 19:54:47.399955 5395 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/master-2/data/info.pb
I20250901 19:54:47.401800 3035 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskaPZSUq/build/asan/bin/kudu as pid 5395
I20250901 19:54:47.409657 5463 sys_catalog.cc:263] Verifying existing consensus state
I20250901 19:54:47.414829 5463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Bootstrap starting.
I20250901 19:54:47.430132 5463 log.cc:826] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Log is configured to *not* fsync() on all Append() calls
I20250901 19:54:47.476786 5463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=56 overwritten=0 applied=56 ignored=0} inserts{seen=32 ignored=0} mutations{seen=39 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 19:54:47.477558 5463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: Bootstrap complete.
I20250901 19:54:47.483367 5463 raft_consensus.cc:357] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:47.484246 5463 raft_consensus.cc:738] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08076b584d2a4aec8fd9868f85c9c7cd, State: Initialized, Role: FOLLOWER
I20250901 19:54:47.485013 5463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 56, Last appended: 8.56, Last appended by leader: 56, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:47.485879 5468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:47.486135 5468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:47.487037 5463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: configured and running, proceeding with master startup.
I20250901 19:54:47.523259 5482 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:47.523514 5482 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: loading cluster ID for follower catalog manager: success
I20250901 19:54:47.528339 5482 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: acquiring CA information for follower catalog manager: success
I20250901 19:54:47.525779 5395 master_runner.cc:186] Error getting master registration for 127.2.246.253:34127: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111),
I20250901 19:54:47.530839 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:54:47.532172 5482 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 19:54:47.572947 3961 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
I20250901 19:54:47.573055 3683 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
I20250901 19:54:47.576913 5428 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" instance_seqno: 1756756457020957) as {username='slave'} at 127.2.246.195:40389; Asking this server to re-register.
I20250901 19:54:47.578356 3823 heartbeater.cc:344] Connected to a master server at 127.2.246.252:37003
I20250901 19:54:47.578416 3961 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:47.578819 5427 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" instance_seqno: 1756756455802456) as {username='slave'} at 127.2.246.193:43525; Asking this server to re-register.
I20250901 19:54:47.579577 3683 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:47.580407 5427 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d397574faadb441192c99d530f14a734" instance_seqno: 1756756456460392) as {username='slave'} at 127.2.246.194:39015; Asking this server to re-register.
I20250901 19:54:47.581147 3823 heartbeater.cc:461] Registering TS with master...
I20250901 19:54:47.581595 5426 ts_manager.cc:194] Registered new tserver with Master: c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:47.581743 5428 ts_manager.cc:194] Registered new tserver with Master: e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:47.582798 5427 ts_manager.cc:194] Registered new tserver with Master: d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:48.028858 5489 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:48.029110 5489 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:48.030851 5489 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
I20250901 19:54:48.031459 5292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 56 } } ignore_live_leader: false dest_uuid: "4c9163569ae1418090a245256e263186" is_pre_election: true
I20250901 19:54:48.031785 5292 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 08076b584d2a4aec8fd9868f85c9c7cd in current term 9: Already voted for candidate 1d944ed54f8c43c9b8c4e7d589dad395 in this term.
W20250901 19:54:48.032051 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:54:48.033525 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:48.033691 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:48.034026 5489 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 8 FOLLOWER]: Advancing to term 9
I20250901 19:54:48.037815 5489 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250901 19:54:48.366076 5493 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:48.366278 5493 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:48.366967 5493 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
W20250901 19:54:48.369612 5268 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:48.374938 5438 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c9163569ae1418090a245256e263186" candidate_term: 10 candidate_status { last_received { term: 9 index: 58 } } ignore_live_leader: false dest_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" is_pre_election: true
I20250901 19:54:48.375376 5438 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c9163569ae1418090a245256e263186 in term 9.
I20250901 19:54:48.376086 5267 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186; no voters: 1d944ed54f8c43c9b8c4e7d589dad395
I20250901 19:54:48.376560 5493 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250901 19:54:48.376691 5493 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1d944ed54f8c43c9b8c4e7d589dad395)
I20250901 19:54:48.376796 5493 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 9 FOLLOWER]: Advancing to term 10
I20250901 19:54:48.378901 5493 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:48.379606 5493 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 election: Requested vote from peers 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127), 08076b584d2a4aec8fd9868f85c9c7cd (127.2.246.252:37003)
I20250901 19:54:48.380270 5438 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c9163569ae1418090a245256e263186" candidate_term: 10 candidate_status { last_received { term: 9 index: 58 } } ignore_live_leader: false dest_uuid: "08076b584d2a4aec8fd9868f85c9c7cd"
I20250901 19:54:48.380517 5438 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 9 FOLLOWER]: Advancing to term 10
W20250901 19:54:48.381017 5268 leader_election.cc:336] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:48.382709 5438 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c9163569ae1418090a245256e263186 in term 10.
I20250901 19:54:48.383278 5267 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd, 4c9163569ae1418090a245256e263186; no voters: 1d944ed54f8c43c9b8c4e7d589dad395
I20250901 19:54:48.383620 5493 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 10 FOLLOWER]: Leader election won for term 10
I20250901 19:54:48.384079 5493 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [term 10 LEADER]: Becoming Leader. State: Replica: 4c9163569ae1418090a245256e263186, State: Running, Role: LEADER
I20250901 19:54:48.384582 5493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 9.58, Last appended by leader: 58, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:48.386231 5498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c9163569ae1418090a245256e263186. Latest consensus state: current_term: 10 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:48.386547 5498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:48.387301 5500 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 19:54:48.390708 5500 catalog_manager.cc:671] Loaded metadata for table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077]
I20250901 19:54:48.391265 5500 catalog_manager.cc:671] Loaded metadata for table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59]
I20250901 19:54:48.391674 5500 catalog_manager.cc:671] Loaded metadata for table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896]
I20250901 19:54:48.392032 5500 catalog_manager.cc:671] Loaded metadata for table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67]
I20250901 19:54:48.392359 5500 catalog_manager.cc:671] Loaded metadata for table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]
I20250901 19:54:48.392722 5500 catalog_manager.cc:671] Loaded metadata for table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612]
I20250901 19:54:48.393041 5500 catalog_manager.cc:671] Loaded metadata for table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8]
I20250901 19:54:48.393410 5500 catalog_manager.cc:671] Loaded metadata for table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616]
I20250901 19:54:48.393743 5500 catalog_manager.cc:671] Loaded metadata for table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6]
I20250901 19:54:48.396517 5500 tablet_loader.cc:96] loaded metadata for tablet 59eb68e59e59483fa9c3999da17aeaec (table default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3])
I20250901 19:54:48.396888 5500 tablet_loader.cc:96] loaded metadata for tablet 5d8e6708fad646d39f3cd20381c1c8aa (table default.table_11ed1535329246fc866753e35e0cb6a1 [id=78b1dcafd0d04873a71930d424cf2d67])
I20250901 19:54:48.397362 5500 tablet_loader.cc:96] loaded metadata for tablet 6579c1fbcfc04dac9345a8157b352591 (table default.table_3cbad8a0189443d9b60b0abc4460862a [id=b2d5ec72c24e4c6f96918db0e2751612])
I20250901 19:54:48.397934 5500 tablet_loader.cc:96] loaded metadata for tablet 8b9fa477370948adb7af0b9d29734522 (table default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616])
I20250901 19:54:48.398289 5500 tablet_loader.cc:96] loaded metadata for tablet 957e86dd3dd940fabcab40043e801e6b (table default.table_d29d54fc138b40009682d5b2585d25bd [id=57841a49107e4375aa82904e212fc896])
I20250901 19:54:48.398741 5500 tablet_loader.cc:96] loaded metadata for tablet b75452cf2414469185c3a763d5a39cac (table default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077])
I20250901 19:54:48.399036 5500 tablet_loader.cc:96] loaded metadata for tablet b867925c426149c99ffdd6d5df396462 (table default.table_0a1ca85b493d4aa285d85b17df52c4aa [id=fe4c5d5ccf384fbfbf05cb7b4f0a1bb6])
I20250901 19:54:48.399354 5500 tablet_loader.cc:96] loaded metadata for tablet e003ca1913284f239f5b0474fc621abe (table default.table_69ed2a785efc4a4f8077c8aa2ef89c35 [id=d9eb1bc0699747928b2a9d2e6fb8bbd8])
I20250901 19:54:48.399750 5500 tablet_loader.cc:96] loaded metadata for tablet e0063b69ff9a48f6a1229cd26704c66d (table default.table_079afb395d5c4c17a18fd7e77548109d [id=513fdaec16fd4268bf9ac4b9cd6cfd59])
I20250901 19:54:48.400813 5500 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 19:54:48.402132 5500 catalog_manager.cc:1261] Loaded cluster ID: 83a9eb5c811b46d48308fced0c9efbb2
I20250901 19:54:48.402223 5500 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 19:54:48.404155 5500 catalog_manager.cc:1506] Loading token signing keys...
I20250901 19:54:48.405273 5500 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Loaded TSK: 0
I20250901 19:54:48.406140 5500 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 19:54:48.408125 5500 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250901 19:54:48.409371 5500 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250901 19:54:48.500140 5281 master_service.cc:948] ReplaceTablet: received request to replace tablet 7b163f22e81940f2a8d9249c5cb64afc from {username='slave'} at 127.0.0.1:50620
I20250901 19:54:48.500460 5282 master_service.cc:948] ReplaceTablet: received request to replace tablet 6579c1fbcfc04dac9345a8157b352591 from {username='slave'} at 127.0.0.1:50620
I20250901 19:54:48.500808 5279 master_service.cc:948] ReplaceTablet: received request to replace tablet 7b163f22e81940f2a8d9249c5cb64afc from {username='slave'} at 127.0.0.1:50620
I20250901 19:54:48.500872 5280 master_service.cc:948] ReplaceTablet: received request to replace tablet e43433b1ce9646e49962fe44bbed2491 from {username='slave'} at 127.0.0.1:50620
I20250901 19:54:48.502378 5281 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50620:
table { table_name: "default.Table_0a1ca85b493d4aa285d85b17df52c4aa" } new_table_name: "default.Table_ed1ee4c8517b4614aafb4f78827e4f6e" modify_external_catalogs: true
I20250901 19:54:48.502658 5279 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50620:
name: "default.Table_14959779c0504cb4bf46faa666221361"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:48.502750 5280 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50620:
name: "default.Table_b7d788fb008945558d1e6fd4dde4f706"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:48.503535 5277 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50620:
name: "default.Table_b679bfff7a2f4d13b5e318b2ec42fc36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 19:54:48.505178 5275 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50620:
name: "default.Table_4c54782dd2bb4c66a3be449970979d0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250901 19:54:48.511732 5340 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_0a1ca85b493d4aa285d85b17df52c4aa, message={"server":"","servicePrincipal":"","db":"default","table":"table_0a1ca85b493d4aa285d85b17df52c4aa","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_0a1ca85b493d4aa285d85b17df52c4aa\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0a1ca85b493d4aa285d85b17df52c4aa\",\"kudu.table_id\":\"fe4c5d5ccf384fbfbf05cb7b4f0a1bb6\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756756457, eventType=CREATE_TABLE, dbName=default, tableName=table_c32254b687594ed89daf3edaded35200, message={"server":"","servicePrincipal":"","db":"default","table":"table_c32254b687594ed89daf3edaded35200","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c32254b687594ed89daf3edaded35200\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756756457},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/warehouse/table_c32254b687594ed89daf3edaded35200\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c32254b687594ed89daf3edaded35200\",\"kudu.table_id\":\"268741b95ddb403f8f59e4219723950e\",\"transient_lastDdlTime\":\"1756756457\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"83a9eb5c811b46d48308fced0c9efbb2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756756457,"files":[]}, messageFormat=json-0.2, catName=hive);
I20250901 19:54:48.511448 5274 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50620:
name: "default.Table_240cfd071a7c475bb44ff628d903bf9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
*** Check failure stack trace: ***
I20250901 19:54:48.511662 5276 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50620:
table { table_name: "default.Table_69ed2a785efc4a4f8077c8aa2ef89c35" } new_table_name: "default.Table_6d9fb4a3f26048cb95788236b2d5ec7f" modify_external_catalogs: true
I20250901 19:54:48.511814 5273 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50620:
table { table_name: "default.Table_51c52d3eec714bbd9a77bed453c02bb1" } modify_external_catalogs: true
I20250901 19:54:48.513142 5273 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50620:
table { table_name: "default.Table_51c52d3eec714bbd9a77bed453c02bb1" } modify_external_catalogs: true
I20250901 19:54:48.510504 5278 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50620:
name: "default.Table_6f60f64babef49459553b6e490e198e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fcc6a2e2dcd google::LogMessage::Fail() at ??:0
@ 0x7fcc6a2e6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fcc6a2e27cc google::LogMessage::Flush() at ??:0
@ 0x7fcc6a2e2a11 google::LogMessage::~LogMessage() at ??:0
I20250901 19:54:48.521514 5438 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Refusing update from remote peer 4c9163569ae1418090a245256e263186: Log matching property violated. Preceding OpId in replica: term: 8 index: 56. Preceding OpId from leader: term: 10 index: 60. (index mismatch)
W20250901 19:54:48.522109 5268 consensus_peers.cc:489] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 -> Peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Couldn't send request to peer 1d944ed54f8c43c9b8c4e7d589dad395. Status: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 19:54:48.522338 5493 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 56, Time since last communication: 0.000s
@ 0x7fcc80717794 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250901 19:54:48.531215 5489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c9163569ae1418090a245256e263186. Latest consensus state: current_term: 10 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:48.531615 5489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:48.531967 5506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:48.532451 5506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
I20250901 19:54:48.535776 5282 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186: Sending DeleteTablet for 3 replicas of tablet 6579c1fbcfc04dac9345a8157b352591
I20250901 19:54:48.536101 5506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:48.536275 5506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fcc80716041 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250901 19:54:48.537417 5510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "4c9163569ae1418090a245256e263186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } } }
I20250901 19:54:48.537786 5510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9163569ae1418090a245256e263186 [sys.catalog]: This master's current role is: LEADER
I20250901 19:54:48.538076 5282 catalog_manager.cc:6652] ReplaceTablet: tablet 6579c1fbcfc04dac9345a8157b352591 deleted and replaced by tablet 8ddc7cee2ee1449196a867bd88e275b7
I20250901 19:54:48.540096 5282 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50620:
table { table_name: "default.Table_5cf2893927ed42d597f83e0a04da2245" } new_table_name: "default.Table_3d911927cc9648b39bc8fae3c7d25c87" modify_external_catalogs: true
@ 0x7fcc8071dcbc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250901 19:54:48.561359 3757 tablet_service.cc:1515] Processing DeleteTablet for tablet 6579c1fbcfc04dac9345a8157b352591 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ddc7cee2ee1449196a867bd88e275b7) from {username='slave'} at 127.0.0.1:36114
@ 0x7fcc8071db39 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250901 19:54:48.568447 3617 tablet_service.cc:1515] Processing DeleteTablet for tablet 6579c1fbcfc04dac9345a8157b352591 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ddc7cee2ee1449196a867bd88e275b7) from {username='slave'} at 127.0.0.1:34746
I20250901 19:54:48.571822 3758 tablet_service.cc:1468] Processing CreateTablet for tablet 8b9fa477370948adb7af0b9d29734522 (DEFAULT_TABLE table=default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:48.572347 3758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9fa477370948adb7af0b9d29734522. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:48.572546 3616 tablet_service.cc:1468] Processing CreateTablet for tablet 8b9fa477370948adb7af0b9d29734522 (DEFAULT_TABLE table=default.table_ea3686ff3ba94b9a984657e263ca0832 [id=e3dcc8f77c32491487c240aeadfb3616]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:48.573073 3616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9fa477370948adb7af0b9d29734522. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fcc6d03fcc9 kudu::Thread::SuperviseThread() at ??:0
I20250901 19:54:48.574681 3615 tablet_service.cc:1468] Processing CreateTablet for tablet 59eb68e59e59483fa9c3999da17aeaec (DEFAULT_TABLE table=default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:48.575088 3615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59eb68e59e59483fa9c3999da17aeaec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:48.576031 3618 tablet_service.cc:1468] Processing CreateTablet for tablet b75452cf2414469185c3a763d5a39cac (DEFAULT_TABLE table=default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7fcc704616db start_thread at ??:0
I20250901 19:54:48.576424 3618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b75452cf2414469185c3a763d5a39cac. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fcc6919371f clone at ??:0
I20250901 19:54:48.579250 3756 tablet_service.cc:1468] Processing CreateTablet for tablet b75452cf2414469185c3a763d5a39cac (DEFAULT_TABLE table=default.table_51c52d3eec714bbd9a77bed453c02bb1 [id=2c6ea8bb63684f33986a43cd87476077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:48.579505 5518 tablet_replica.cc:331] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734: stopping tablet replica
I20250901 19:54:48.579752 5518 raft_consensus.cc:2241] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:48.579809 3756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b75452cf2414469185c3a763d5a39cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:48.579989 5518 raft_consensus.cc:2270] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:48.580636 5518 ts_tablet_manager.cc:1905] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 19:54:48.582202 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250901 19:54:48.582257 3755 tablet_service.cc:1468] Processing CreateTablet for tablet 59eb68e59e59483fa9c3999da17aeaec (DEFAULT_TABLE table=default.table_5cf2893927ed42d597f83e0a04da2245 [id=9a7c50e73c39409f82c0c4339e7af8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 19:54:48.582762 3755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59eb68e59e59483fa9c3999da17aeaec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:48.586990 5521 tablet_replica.cc:331] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21: stopping tablet replica
I20250901 19:54:48.587201 5521 raft_consensus.cc:2241] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 19:54:48.587441 5521 raft_consensus.cc:2270] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 19:54:48.589042 5521 ts_tablet_manager.cc:1905] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 19:54:48.591049 5518 ts_tablet_manager.cc:1918] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 19:54:48.591199 5518 log.cc:1199] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-1/wal/wals/6579c1fbcfc04dac9345a8157b352591
I20250901 19:54:48.591737 5518 ts_tablet_manager.cc:1939] T 6579c1fbcfc04dac9345a8157b352591 P d397574faadb441192c99d530f14a734: Deleting consensus metadata
I20250901 19:54:48.596729 5522 tablet_bootstrap.cc:492] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:48.600638 5524 tablet_bootstrap.cc:492] T 8b9fa477370948adb7af0b9d29734522 P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:48.600715 5522 tablet_bootstrap.cc:654] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:48.602851 5524 tablet_bootstrap.cc:654] T 8b9fa477370948adb7af0b9d29734522 P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:48.606685 5521 ts_tablet_manager.cc:1918] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 19:54:48.606791 5522 tablet_bootstrap.cc:492] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:48.606833 5521 log.cc:1199] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21: Deleting WAL directory at /tmp/dist-test-taskaPZSUq/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756756437511339-3035-0/minicluster-data/ts-0/wal/wals/6579c1fbcfc04dac9345a8157b352591
I20250901 19:54:48.607306 5521 ts_tablet_manager.cc:1939] T 6579c1fbcfc04dac9345a8157b352591 P c5b48d93671f4eb8980759b876e9ca21: Deleting consensus metadata
I20250901 19:54:48.607410 5522 ts_tablet_manager.cc:1397] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250901 19:54:48.608243 5522 raft_consensus.cc:357] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:48.608465 5522 raft_consensus.cc:383] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:48.608567 5522 raft_consensus.cc:738] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:48.608834 5522 consensus_queue.cc:260] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:48.609563 5522 ts_tablet_manager.cc:1428] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 19:54:48.609869 5522 tablet_bootstrap.cc:492] T 59eb68e59e59483fa9c3999da17aeaec P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:48.611682 5522 tablet_bootstrap.cc:654] T 59eb68e59e59483fa9c3999da17aeaec P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:48.621534 5524 tablet_bootstrap.cc:492] T 8b9fa477370948adb7af0b9d29734522 P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:48.621713 5524 ts_tablet_manager.cc:1397] T 8b9fa477370948adb7af0b9d29734522 P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.007s
I20250901 19:54:48.622571 5524 raft_consensus.cc:357] T 8b9fa477370948adb7af0b9d29734522 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:48.622778 5524 raft_consensus.cc:383] T 8b9fa477370948adb7af0b9d29734522 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:48.622867 5524 raft_consensus.cc:738] T 8b9fa477370948adb7af0b9d29734522 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:48.623137 5524 consensus_queue.cc:260] T 8b9fa477370948adb7af0b9d29734522 P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:48.623780 5524 ts_tablet_manager.cc:1428] T 8b9fa477370948adb7af0b9d29734522 P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250901 19:54:48.624173 5524 tablet_bootstrap.cc:492] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:48.624650 3684 heartbeater.cc:499] Master 127.2.246.254:33523 was elected leader, sending a full tablet report...
I20250901 19:54:48.626019 5524 tablet_bootstrap.cc:654] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:48.628065 5524 tablet_bootstrap.cc:492] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:48.628212 5524 ts_tablet_manager.cc:1397] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250901 19:54:48.628999 5524 raft_consensus.cc:357] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:48.629181 5524 raft_consensus.cc:383] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:48.629261 5524 raft_consensus.cc:738] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:48.629338 5522 tablet_bootstrap.cc:492] T 59eb68e59e59483fa9c3999da17aeaec P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:48.629518 5522 ts_tablet_manager.cc:1397] T 59eb68e59e59483fa9c3999da17aeaec P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.008s
I20250901 19:54:48.629561 5524 consensus_queue.cc:260] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:48.630368 5522 raft_consensus.cc:357] T 59eb68e59e59483fa9c3999da17aeaec P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:48.630646 5522 raft_consensus.cc:383] T 59eb68e59e59483fa9c3999da17aeaec P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:48.630750 5522 raft_consensus.cc:738] T 59eb68e59e59483fa9c3999da17aeaec P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:48.631445 5524 ts_tablet_manager.cc:1428] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 19:54:48.631807 5524 tablet_bootstrap.cc:492] T b75452cf2414469185c3a763d5a39cac P c5b48d93671f4eb8980759b876e9ca21: Bootstrap starting.
I20250901 19:54:48.633452 5522 consensus_queue.cc:260] T 59eb68e59e59483fa9c3999da17aeaec P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:48.633611 5524 tablet_bootstrap.cc:654] T b75452cf2414469185c3a763d5a39cac P c5b48d93671f4eb8980759b876e9ca21: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:48.635169 5522 ts_tablet_manager.cc:1428] T 59eb68e59e59483fa9c3999da17aeaec P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250901 19:54:48.636373 5522 tablet_bootstrap.cc:492] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734: Bootstrap starting.
I20250901 19:54:48.638175 5522 tablet_bootstrap.cc:654] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734: Neither blocks nor log segments found. Creating new log.
I20250901 19:54:48.640918 5522 tablet_bootstrap.cc:492] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734: No bootstrap required, opened a new log
I20250901 19:54:48.641026 5522 ts_tablet_manager.cc:1397] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250901 19:54:48.641176 5524 tablet_bootstrap.cc:492] T b75452cf2414469185c3a763d5a39cac P c5b48d93671f4eb8980759b876e9ca21: No bootstrap required, opened a new log
I20250901 19:54:48.641345 5524 ts_tablet_manager.cc:1397] T b75452cf2414469185c3a763d5a39cac P c5b48d93671f4eb8980759b876e9ca21: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20250901 19:54:48.641583 5522 raft_consensus.cc:357] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:48.641710 5522 raft_consensus.cc:383] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:48.641765 5522 raft_consensus.cc:738] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d397574faadb441192c99d530f14a734, State: Initialized, Role: FOLLOWER
I20250901 19:54:48.641991 5522 consensus_queue.cc:260] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:48.642131 5524 raft_consensus.cc:357] T b75452cf2414469185c3a763d5a39cac P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:48.642308 5524 raft_consensus.cc:383] T b75452cf2414469185c3a763d5a39cac P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 19:54:48.642482 5524 raft_consensus.cc:738] T b75452cf2414469185c3a763d5a39cac P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Initialized, Role: FOLLOWER
I20250901 19:54:48.642583 5522 ts_tablet_manager.cc:1428] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250901 19:54:48.642732 5524 consensus_queue.cc:260] T b75452cf2414469185c3a763d5a39cac P c5b48d93671f4eb8980759b876e9ca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:48.644220 5524 ts_tablet_manager.cc:1428] T b75452cf2414469185c3a763d5a39cac P c5b48d93671f4eb8980759b876e9ca21: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20250901 19:54:48.650274 3686 tablet.cc:2378] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250901 19:54:48.790753 3572 connection.cc:537] server connection from 127.0.0.1:34746 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:48.790753 3714 connection.cc:537] server connection from 127.0.0.1:36114 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:48.791137 5514 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 19:54:48.539707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43450 (local address 127.2.246.195:33271)
0901 19:54:48.568120 (+ 28413us) server_negotiation.cc:206] Beginning negotiation
0901 19:54:48.568129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0901 19:54:48.568154 (+ 25us) server_negotiation.cc:407] Connection header received
0901 19:54:48.568235 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 19:54:48.568243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 19:54:48.568351 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 19:54:48.568481 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 19:54:48.790823 (+222342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43450: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":28255,"thread_start_us":28893,"threads_started":1}
W20250901 19:54:48.791138 3573 connection.cc:537] client connection to 127.2.246.254:33523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:48.791612 3713 connection.cc:537] client connection to 127.2.246.254:33523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:48.791975 3973 connection.cc:537] client connection to 127.2.246.254:33523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 19:54:48.792634 3973 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.246.254:33523)
W20250901 19:54:48.793232 3973 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.246.254:33523)
W20250901 19:54:48.793452 3973 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.246.254:33523)
W20250901 19:54:48.794191 3973 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.246.254:33523)
W20250901 19:54:48.849668 3831 tablet.cc:2378] T 59eb68e59e59483fa9c3999da17aeaec P d397574faadb441192c99d530f14a734: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 19:54:48.855593 5488 raft_consensus.cc:491] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:48.855840 5488 raft_consensus.cc:513] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:48.856664 5488 leader_election.cc:290] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:48.857146 3637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b75452cf2414469185c3a763d5a39cac" candidate_uuid: "d397574faadb441192c99d530f14a734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21" is_pre_election: true
I20250901 19:54:48.857373 3637 raft_consensus.cc:2466] T b75452cf2414469185c3a763d5a39cac P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d397574faadb441192c99d530f14a734 in term 0.
W20250901 19:54:48.857676 3712 leader_election.cc:343] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271): Not found: Tablet not found: b75452cf2414469185c3a763d5a39cac
I20250901 19:54:48.857878 3712 leader_election.cc:304] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters: e0064a1b18e5470b85bde2277757aac8
I20250901 19:54:48.858150 5488 raft_consensus.cc:2802] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:48.858263 5488 raft_consensus.cc:491] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:48.858352 5488 raft_consensus.cc:3058] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:48.860328 5488 raft_consensus.cc:513] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:48.861069 5488 leader_election.cc:290] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 election: Requested vote from peers c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:48.861588 3637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b75452cf2414469185c3a763d5a39cac" candidate_uuid: "d397574faadb441192c99d530f14a734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21"
I20250901 19:54:48.861811 3637 raft_consensus.cc:3058] T b75452cf2414469185c3a763d5a39cac P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
W20250901 19:54:48.861915 3712 leader_election.cc:343] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271): Not found: Tablet not found: b75452cf2414469185c3a763d5a39cac
I20250901 19:54:48.863838 3637 raft_consensus.cc:2466] T b75452cf2414469185c3a763d5a39cac P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d397574faadb441192c99d530f14a734 in term 1.
I20250901 19:54:48.864233 3711 leader_election.cc:304] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters: e0064a1b18e5470b85bde2277757aac8
I20250901 19:54:48.864565 5488 raft_consensus.cc:2802] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:48.864948 5488 raft_consensus.cc:695] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [term 1 LEADER]: Becoming Leader. State: Replica: d397574faadb441192c99d530f14a734, State: Running, Role: LEADER
I20250901 19:54:48.865274 5488 consensus_queue.cc:237] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:48.866660 5384 raft_consensus.cc:491] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:48.866813 5384 raft_consensus.cc:513] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:48.867584 5384 leader_election.cc:290] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d397574faadb441192c99d530f14a734 (127.2.246.194:33557), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:48.869005 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59eb68e59e59483fa9c3999da17aeaec" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734" is_pre_election: true
I20250901 19:54:48.869197 3778 raft_consensus.cc:2466] T 59eb68e59e59483fa9c3999da17aeaec P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 0.
W20250901 19:54:48.869565 3572 leader_election.cc:343] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271): Not found: Tablet not found: 59eb68e59e59483fa9c3999da17aeaec
I20250901 19:54:48.869776 3573 leader_election.cc:304] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters: e0064a1b18e5470b85bde2277757aac8
I20250901 19:54:48.870052 5384 raft_consensus.cc:2802] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:48.870170 5384 raft_consensus.cc:491] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:48.870251 5384 raft_consensus.cc:3058] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:48.872184 5384 raft_consensus.cc:513] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:48.872740 5384 leader_election.cc:290] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Requested vote from peers d397574faadb441192c99d530f14a734 (127.2.246.194:33557), e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271)
I20250901 19:54:48.873196 3778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59eb68e59e59483fa9c3999da17aeaec" candidate_uuid: "c5b48d93671f4eb8980759b876e9ca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d397574faadb441192c99d530f14a734"
I20250901 19:54:48.873438 3778 raft_consensus.cc:3058] T 59eb68e59e59483fa9c3999da17aeaec P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
W20250901 19:54:48.873759 3572 leader_election.cc:343] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271): Not found: Tablet not found: 59eb68e59e59483fa9c3999da17aeaec
I20250901 19:54:48.875659 3778 raft_consensus.cc:2466] T 59eb68e59e59483fa9c3999da17aeaec P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b48d93671f4eb8980759b876e9ca21 in term 1.
I20250901 19:54:48.876118 3573 leader_election.cc:304] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters: e0064a1b18e5470b85bde2277757aac8
I20250901 19:54:48.876420 5384 raft_consensus.cc:2802] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:48.876853 5384 raft_consensus.cc:695] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [term 1 LEADER]: Becoming Leader. State: Replica: c5b48d93671f4eb8980759b876e9ca21, State: Running, Role: LEADER
I20250901 19:54:48.877190 5384 consensus_queue.cc:237] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:48.937148 5488 raft_consensus.cc:491] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 19:54:48.937435 5488 raft_consensus.cc:513] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:48.938112 5488 leader_election.cc:290] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271), c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:48.938757 3637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b9fa477370948adb7af0b9d29734522" candidate_uuid: "d397574faadb441192c99d530f14a734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21" is_pre_election: true
I20250901 19:54:48.938992 3637 raft_consensus.cc:2466] T 8b9fa477370948adb7af0b9d29734522 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d397574faadb441192c99d530f14a734 in term 0.
W20250901 19:54:48.939141 3712 leader_election.cc:343] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271): Not found: Tablet not found: 8b9fa477370948adb7af0b9d29734522
I20250901 19:54:48.939399 3711 leader_election.cc:304] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters: e0064a1b18e5470b85bde2277757aac8
I20250901 19:54:48.939806 5488 raft_consensus.cc:2802] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 19:54:48.939924 5488 raft_consensus.cc:491] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 19:54:48.940032 5488 raft_consensus.cc:3058] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:48.941936 5488 raft_consensus.cc:513] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:48.942642 5488 leader_election.cc:290] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 election: Requested vote from peers e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271), c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:48.943190 3637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b9fa477370948adb7af0b9d29734522" candidate_uuid: "d397574faadb441192c99d530f14a734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b48d93671f4eb8980759b876e9ca21"
W20250901 19:54:48.943409 3712 leader_election.cc:343] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271): Not found: Tablet not found: 8b9fa477370948adb7af0b9d29734522
I20250901 19:54:48.943473 3637 raft_consensus.cc:3058] T 8b9fa477370948adb7af0b9d29734522 P c5b48d93671f4eb8980759b876e9ca21 [term 0 FOLLOWER]: Advancing to term 1
I20250901 19:54:48.945608 3637 raft_consensus.cc:2466] T 8b9fa477370948adb7af0b9d29734522 P c5b48d93671f4eb8980759b876e9ca21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d397574faadb441192c99d530f14a734 in term 1.
I20250901 19:54:48.946029 3711 leader_election.cc:304] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5b48d93671f4eb8980759b876e9ca21, d397574faadb441192c99d530f14a734; no voters: e0064a1b18e5470b85bde2277757aac8
I20250901 19:54:48.946404 5488 raft_consensus.cc:2802] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 19:54:48.946561 5488 raft_consensus.cc:695] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [term 1 LEADER]: Becoming Leader. State: Replica: d397574faadb441192c99d530f14a734, State: Running, Role: LEADER
I20250901 19:54:48.946859 5488 consensus_queue.cc:237] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:48.983001 3962 heartbeater.cc:499] Master 127.2.246.254:33523 was elected leader, sending a full tablet report...
I20250901 19:54:49.070760 5543 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:54:49.071005 5543 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:49.071972 5543 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:49.074771 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:49.075011 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:49.075109 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:49.075366 5543 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250901 19:54:49.300779 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 1: this message will repeat every 5th retry.
I20250901 19:54:49.634392 5556 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:54:49.634653 5556 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:49.635315 5556 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:49.637604 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:49.637821 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:49.637931 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:49.638223 5556 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:54:49.732599 5384 consensus_queue.cc:1035] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 19:54:49.739415 3572 consensus_peers.cc:489] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21 -> Peer e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271): Couldn't send request to peer e0064a1b18e5470b85bde2277757aac8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59eb68e59e59483fa9c3999da17aeaec. This is attempt 1: this message will repeat every 5th retry.
I20250901 19:54:49.826395 5555 consensus_queue.cc:1035] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 19:54:49.831679 3712 consensus_peers.cc:489] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734 -> Peer e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271): Couldn't send request to peer e0064a1b18e5470b85bde2277757aac8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b75452cf2414469185c3a763d5a39cac. This is attempt 1: this message will repeat every 5th retry.
I20250901 19:54:49.948396 3035 master-stress-test.cc:577] Killing test thread 1/25
W20250901 19:54:50.110491 3712 consensus_peers.cc:489] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 -> Peer e0064a1b18e5470b85bde2277757aac8 (127.2.246.195:33271): Couldn't send request to peer e0064a1b18e5470b85bde2277757aac8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b9fa477370948adb7af0b9d29734522. This is attempt 1: this message will repeat every 5th retry.
I20250901 19:54:50.148077 5555 consensus_queue.cc:1035] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 19:54:50.209748 5570 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:54:50.209970 5570 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:50.210640 5570 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:50.213042 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:50.213274 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:50.213375 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:50.213696 5570 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:54:50.513113 5579 ts_tablet_manager.cc:927] T 59eb68e59e59483fa9c3999da17aeaec P e0064a1b18e5470b85bde2277757aac8: Initiating tablet copy from peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767)
I20250901 19:54:50.514145 5579 tablet_copy_client.cc:323] T 59eb68e59e59483fa9c3999da17aeaec P e0064a1b18e5470b85bde2277757aac8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.246.193:37767
I20250901 19:54:50.515414 3658 tablet_copy_service.cc:140] P c5b48d93671f4eb8980759b876e9ca21: Received BeginTabletCopySession request for tablet 59eb68e59e59483fa9c3999da17aeaec from peer e0064a1b18e5470b85bde2277757aac8 ({username='slave'} at 127.2.246.195:46933)
I20250901 19:54:50.515679 3658 tablet_copy_service.cc:161] P c5b48d93671f4eb8980759b876e9ca21: Beginning new tablet copy session on tablet 59eb68e59e59483fa9c3999da17aeaec from peer e0064a1b18e5470b85bde2277757aac8 at {username='slave'} at 127.2.246.195:46933: session id = e0064a1b18e5470b85bde2277757aac8-59eb68e59e59483fa9c3999da17aeaec
I20250901 19:54:50.518303 3658 tablet_copy_source_session.cc:215] T 59eb68e59e59483fa9c3999da17aeaec P c5b48d93671f4eb8980759b876e9ca21: Tablet Copy: opened 0 blocks and 1 log segments
I20250901 19:54:50.520498 5579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59eb68e59e59483fa9c3999da17aeaec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:50.525074 5579 tablet_copy_client.cc:806] T 59eb68e59e59483fa9c3999da17aeaec P e0064a1b18e5470b85bde2277757aac8: tablet copy: Starting download of 0 data blocks...
I20250901 19:54:50.525359 5579 tablet_copy_client.cc:670] T 59eb68e59e59483fa9c3999da17aeaec P e0064a1b18e5470b85bde2277757aac8: tablet copy: Starting download of 1 WAL segments...
I20250901 19:54:50.527926 5579 tablet_copy_client.cc:538] T 59eb68e59e59483fa9c3999da17aeaec P e0064a1b18e5470b85bde2277757aac8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250901 19:54:50.530929 5579 tablet_bootstrap.cc:492] T 59eb68e59e59483fa9c3999da17aeaec P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:50.537075 5579 tablet_bootstrap.cc:492] T 59eb68e59e59483fa9c3999da17aeaec P e0064a1b18e5470b85bde2277757aac8: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 19:54:50.537460 5579 tablet_bootstrap.cc:492] T 59eb68e59e59483fa9c3999da17aeaec P e0064a1b18e5470b85bde2277757aac8: Bootstrap complete.
I20250901 19:54:50.537664 5579 ts_tablet_manager.cc:1397] T 59eb68e59e59483fa9c3999da17aeaec P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.008s
I20250901 19:54:50.538312 5579 raft_consensus.cc:357] T 59eb68e59e59483fa9c3999da17aeaec P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:50.538471 5579 raft_consensus.cc:738] T 59eb68e59e59483fa9c3999da17aeaec P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:50.538668 5579 consensus_queue.cc:260] T 59eb68e59e59483fa9c3999da17aeaec P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:50.539428 5579 ts_tablet_manager.cc:1428] T 59eb68e59e59483fa9c3999da17aeaec P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250901 19:54:50.540360 3658 tablet_copy_service.cc:342] P c5b48d93671f4eb8980759b876e9ca21: Request end of tablet copy session e0064a1b18e5470b85bde2277757aac8-59eb68e59e59483fa9c3999da17aeaec received from {username='slave'} at 127.2.246.195:46933
I20250901 19:54:50.540550 3658 tablet_copy_service.cc:434] P c5b48d93671f4eb8980759b876e9ca21: ending tablet copy session e0064a1b18e5470b85bde2277757aac8-59eb68e59e59483fa9c3999da17aeaec on tablet 59eb68e59e59483fa9c3999da17aeaec with peer e0064a1b18e5470b85bde2277757aac8
I20250901 19:54:50.937232 5579 ts_tablet_manager.cc:927] T 8b9fa477370948adb7af0b9d29734522 P e0064a1b18e5470b85bde2277757aac8: Initiating tablet copy from peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:50.937901 5579 tablet_copy_client.cc:323] T 8b9fa477370948adb7af0b9d29734522 P e0064a1b18e5470b85bde2277757aac8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.246.194:33557
I20250901 19:54:50.938944 3798 tablet_copy_service.cc:140] P d397574faadb441192c99d530f14a734: Received BeginTabletCopySession request for tablet 8b9fa477370948adb7af0b9d29734522 from peer e0064a1b18e5470b85bde2277757aac8 ({username='slave'} at 127.2.246.195:52841)
I20250901 19:54:50.939147 3798 tablet_copy_service.cc:161] P d397574faadb441192c99d530f14a734: Beginning new tablet copy session on tablet 8b9fa477370948adb7af0b9d29734522 from peer e0064a1b18e5470b85bde2277757aac8 at {username='slave'} at 127.2.246.195:52841: session id = e0064a1b18e5470b85bde2277757aac8-8b9fa477370948adb7af0b9d29734522
I20250901 19:54:50.941720 3798 tablet_copy_source_session.cc:215] T 8b9fa477370948adb7af0b9d29734522 P d397574faadb441192c99d530f14a734: Tablet Copy: opened 0 blocks and 1 log segments
I20250901 19:54:50.943295 5579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9fa477370948adb7af0b9d29734522. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:50.947660 5579 tablet_copy_client.cc:806] T 8b9fa477370948adb7af0b9d29734522 P e0064a1b18e5470b85bde2277757aac8: tablet copy: Starting download of 0 data blocks...
I20250901 19:54:50.947924 5579 tablet_copy_client.cc:670] T 8b9fa477370948adb7af0b9d29734522 P e0064a1b18e5470b85bde2277757aac8: tablet copy: Starting download of 1 WAL segments...
I20250901 19:54:50.950001 5579 tablet_copy_client.cc:538] T 8b9fa477370948adb7af0b9d29734522 P e0064a1b18e5470b85bde2277757aac8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250901 19:54:50.952556 5579 tablet_bootstrap.cc:492] T 8b9fa477370948adb7af0b9d29734522 P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:50.956772 5579 tablet_bootstrap.cc:492] T 8b9fa477370948adb7af0b9d29734522 P e0064a1b18e5470b85bde2277757aac8: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 19:54:50.957134 5579 tablet_bootstrap.cc:492] T 8b9fa477370948adb7af0b9d29734522 P e0064a1b18e5470b85bde2277757aac8: Bootstrap complete.
I20250901 19:54:50.957362 5579 ts_tablet_manager.cc:1397] T 8b9fa477370948adb7af0b9d29734522 P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250901 19:54:50.957983 5579 raft_consensus.cc:357] T 8b9fa477370948adb7af0b9d29734522 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:50.958191 5579 raft_consensus.cc:738] T 8b9fa477370948adb7af0b9d29734522 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:50.958469 5579 consensus_queue.cc:260] T 8b9fa477370948adb7af0b9d29734522 P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } }
I20250901 19:54:50.958937 5579 ts_tablet_manager.cc:1428] T 8b9fa477370948adb7af0b9d29734522 P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250901 19:54:50.960301 3798 tablet_copy_service.cc:342] P d397574faadb441192c99d530f14a734: Request end of tablet copy session e0064a1b18e5470b85bde2277757aac8-8b9fa477370948adb7af0b9d29734522 received from {username='slave'} at 127.2.246.195:52841
I20250901 19:54:50.960475 3798 tablet_copy_service.cc:434] P d397574faadb441192c99d530f14a734: ending tablet copy session e0064a1b18e5470b85bde2277757aac8-8b9fa477370948adb7af0b9d29734522 on tablet 8b9fa477370948adb7af0b9d29734522 with peer e0064a1b18e5470b85bde2277757aac8
I20250901 19:54:51.007465 5579 ts_tablet_manager.cc:927] T b75452cf2414469185c3a763d5a39cac P e0064a1b18e5470b85bde2277757aac8: Initiating tablet copy from peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557)
I20250901 19:54:51.008133 5579 tablet_copy_client.cc:323] T b75452cf2414469185c3a763d5a39cac P e0064a1b18e5470b85bde2277757aac8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.246.194:33557
I20250901 19:54:51.008863 3798 tablet_copy_service.cc:140] P d397574faadb441192c99d530f14a734: Received BeginTabletCopySession request for tablet b75452cf2414469185c3a763d5a39cac from peer e0064a1b18e5470b85bde2277757aac8 ({username='slave'} at 127.2.246.195:52841)
I20250901 19:54:51.009050 3798 tablet_copy_service.cc:161] P d397574faadb441192c99d530f14a734: Beginning new tablet copy session on tablet b75452cf2414469185c3a763d5a39cac from peer e0064a1b18e5470b85bde2277757aac8 at {username='slave'} at 127.2.246.195:52841: session id = e0064a1b18e5470b85bde2277757aac8-b75452cf2414469185c3a763d5a39cac
I20250901 19:54:51.010952 3798 tablet_copy_source_session.cc:215] T b75452cf2414469185c3a763d5a39cac P d397574faadb441192c99d530f14a734: Tablet Copy: opened 0 blocks and 1 log segments
I20250901 19:54:51.012164 5579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b75452cf2414469185c3a763d5a39cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 19:54:51.016549 5579 tablet_copy_client.cc:806] T b75452cf2414469185c3a763d5a39cac P e0064a1b18e5470b85bde2277757aac8: tablet copy: Starting download of 0 data blocks...
I20250901 19:54:51.016801 5579 tablet_copy_client.cc:670] T b75452cf2414469185c3a763d5a39cac P e0064a1b18e5470b85bde2277757aac8: tablet copy: Starting download of 1 WAL segments...
I20250901 19:54:51.018481 5579 tablet_copy_client.cc:538] T b75452cf2414469185c3a763d5a39cac P e0064a1b18e5470b85bde2277757aac8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250901 19:54:51.021081 5579 tablet_bootstrap.cc:492] T b75452cf2414469185c3a763d5a39cac P e0064a1b18e5470b85bde2277757aac8: Bootstrap starting.
I20250901 19:54:51.025573 5579 tablet_bootstrap.cc:492] T b75452cf2414469185c3a763d5a39cac P e0064a1b18e5470b85bde2277757aac8: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 19:54:51.025942 5579 tablet_bootstrap.cc:492] T b75452cf2414469185c3a763d5a39cac P e0064a1b18e5470b85bde2277757aac8: Bootstrap complete.
I20250901 19:54:51.026171 5579 ts_tablet_manager.cc:1397] T b75452cf2414469185c3a763d5a39cac P e0064a1b18e5470b85bde2277757aac8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20250901 19:54:51.026826 5579 raft_consensus.cc:357] T b75452cf2414469185c3a763d5a39cac P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:51.026988 5579 raft_consensus.cc:738] T b75452cf2414469185c3a763d5a39cac P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0064a1b18e5470b85bde2277757aac8, State: Initialized, Role: FOLLOWER
I20250901 19:54:51.027232 5579 consensus_queue.cc:260] T b75452cf2414469185c3a763d5a39cac P e0064a1b18e5470b85bde2277757aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d397574faadb441192c99d530f14a734" member_type: VOTER last_known_addr { host: "127.2.246.194" port: 33557 } } peers { permanent_uuid: "c5b48d93671f4eb8980759b876e9ca21" member_type: VOTER last_known_addr { host: "127.2.246.193" port: 37767 } } peers { permanent_uuid: "e0064a1b18e5470b85bde2277757aac8" member_type: VOTER last_known_addr { host: "127.2.246.195" port: 33271 } }
I20250901 19:54:51.028376 5579 ts_tablet_manager.cc:1428] T b75452cf2414469185c3a763d5a39cac P e0064a1b18e5470b85bde2277757aac8: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250901 19:54:51.029317 3798 tablet_copy_service.cc:342] P d397574faadb441192c99d530f14a734: Request end of tablet copy session e0064a1b18e5470b85bde2277757aac8-b75452cf2414469185c3a763d5a39cac received from {username='slave'} at 127.2.246.195:52841
I20250901 19:54:51.029459 3798 tablet_copy_service.cc:434] P d397574faadb441192c99d530f14a734: ending tablet copy session e0064a1b18e5470b85bde2277757aac8-b75452cf2414469185c3a763d5a39cac on tablet b75452cf2414469185c3a763d5a39cac with peer e0064a1b18e5470b85bde2277757aac8
I20250901 19:54:51.548952 3914 raft_consensus.cc:1215] T 59eb68e59e59483fa9c3999da17aeaec P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250901 19:54:51.754762 5598 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:54:51.754959 5598 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:51.755617 5598 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:51.758006 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:51.758201 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:51.758297 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:51.758538 5598 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:54:52.101897 3914 raft_consensus.cc:1215] T 8b9fa477370948adb7af0b9d29734522 P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250901 19:54:52.208523 3914 raft_consensus.cc:1215] T b75452cf2414469185c3a763d5a39cac P e0064a1b18e5470b85bde2277757aac8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250901 19:54:52.532516 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:54:52.532737 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:54:53.277413 5623 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:54:53.277621 5623 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:53.278280 5623 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:53.279327 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20250901 19:54:53.280786 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:53.280972 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:53.281106 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:53.281392 5623 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:54:54.449764 5640 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:54:54.449978 5640 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:54.450656 5640 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:54.453894 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:54.454135 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:54.454224 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:54.454653 5640 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:54:57.099174 5673 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:54:57.099380 5673 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:54:57.100174 5673 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:54:57.102540 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:54:57.102739 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:54:57.102833 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:54:57.103109 5673 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:54:57.534049 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:54:57.534260 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:54:58.392100 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 6: this message will repeat every 5th retry.
W20250901 19:54:59.061628 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 6: this message will repeat every 5th retry.
I20250901 19:55:02.535887 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:55:02.536116 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:55:05.422251 5788 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:55:05.422464 5788 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:55:05.423206 5788 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:55:05.424592 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250901 19:55:05.426095 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:55:05.426290 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:55:05.426415 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:55:05.426680 5788 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:55:07.538650 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:55:07.538865 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:55:08.805109 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 11: this message will repeat every 5th retry.
W20250901 19:55:08.896394 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 11: this message will repeat every 5th retry.
I20250901 19:55:12.540439 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:55:12.540649 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:55:14.280762 5901 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:55:14.281122 5901 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:55:14.281883 5901 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:55:14.283058 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:55:14.284490 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:55:14.284724 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:55:14.284853 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:55:14.285141 5901 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:55:17.542239 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:55:17.542454 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:55:17.643134 3684 heartbeater.cc:646] Failed to heartbeat to 127.2.246.254:33523 (64 consecutive failures): Network error: Failed to ping master at 127.2.246.254:33523: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:55:17.800891 3829 heartbeater.cc:646] Failed to heartbeat to 127.2.246.253:34127 (78 consecutive failures): Network error: Failed to ping master at 127.2.246.253:34127: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:55:18.035344 3962 heartbeater.cc:646] Failed to heartbeat to 127.2.246.254:33523 (92 consecutive failures): Network error: Failed to ping master at 127.2.246.254:33523: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:55:18.616400 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 16: this message will repeat every 5th retry.
I20250901 19:55:19.526203 5961 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:55:19.526408 5961 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:55:19.527086 5961 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:55:19.528570 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:55:19.530094 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:55:19.530274 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:55:19.530386 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:55:19.530669 5961 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250901 19:55:19.616811 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 16: this message will repeat every 5th retry.
I20250901 19:55:22.544131 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:55:22.544371 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:55:27.545882 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:55:27.546104 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:55:28.849541 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 21: this message will repeat every 5th retry.
W20250901 19:55:30.697098 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 21: this message will repeat every 5th retry.
I20250901 19:55:31.781726 6133 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:55:31.782055 6133 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:55:31.782735 6133 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:55:31.783814 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:55:31.785229 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:55:31.785360 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:55:31.785440 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:55:31.785630 6133 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:55:32.547485 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:55:32.547744 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:55:37.549270 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:55:37.549466 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:55:39.341187 6232 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:55:39.341415 6232 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:55:39.342070 6232 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:55:39.343290 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:55:39.345155 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:55:39.345359 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:55:39.345470 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:55:39.345727 6232 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250901 19:55:39.553664 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 26: this message will repeat every 5th retry.
W20250901 19:55:40.505020 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 26: this message will repeat every 5th retry.
I20250901 19:55:42.551088 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:55:42.551330 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:55:46.369773 6334 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:55:46.370018 6334 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:55:46.370834 6334 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:55:46.372143 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:55:46.373551 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:55:46.373746 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:55:46.373857 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:55:46.374125 6334 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:55:47.552860 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:55:47.553085 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:55:49.224818 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 31: this message will repeat every 5th retry.
W20250901 19:55:50.835952 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 31: this message will repeat every 5th retry.
I20250901 19:55:52.554656 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:55:52.554872 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:55:55.629365 6450 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:55:55.629571 6450 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:55:55.630193 6450 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:55:55.631353 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:55:55.632735 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:55:55.632929 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:55:55.633040 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:55:55.633303 6450 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:55:57.556488 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:55:57.556703 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:55:59.073726 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 36: this message will repeat every 5th retry.
I20250901 19:56:00.484565 6519 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:56:00.484786 6519 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:56:00.485525 6519 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:56:00.488029 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:56:00.488225 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:56:00.488338 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:56:00.488579 6519 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250901 19:56:00.679713 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 36: this message will repeat every 5th retry.
I20250901 19:56:02.558292 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:56:02.558490 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:56:06.635226 6605 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:56:06.635448 6605 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:56:06.636197 6605 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:56:06.637370 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250901 19:56:06.638752 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:56:06.638943 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:56:06.639164 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:56:06.639495 6605 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:56:07.559947 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:56:07.560137 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:56:09.468109 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 41: this message will repeat every 5th retry.
W20250901 19:56:10.883950 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 41: this message will repeat every 5th retry.
I20250901 19:56:12.561661 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:56:12.561883 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:56:17.563511 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:56:17.563750 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:56:17.849319 3684 heartbeater.cc:646] Failed to heartbeat to 127.2.246.254:33523 (184 consecutive failures): Network error: Failed to ping master at 127.2.246.254:33523: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:56:18.002146 3829 heartbeater.cc:646] Failed to heartbeat to 127.2.246.253:34127 (198 consecutive failures): Network error: Failed to ping master at 127.2.246.253:34127: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:56:18.470629 3963 heartbeater.cc:646] Failed to heartbeat to 127.2.246.253:34127 (282 consecutive failures): Network error: Failed to ping master at 127.2.246.253:34127: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:56:19.628661 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 46: this message will repeat every 5th retry.
W20250901 19:56:21.358731 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 46: this message will repeat every 5th retry.
I20250901 19:56:22.565307 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:56:22.565506 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:56:26.346072 6866 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:56:26.346290 6866 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:56:26.347110 6866 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:56:26.348423 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:56:26.352818 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:56:26.353042 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:56:26.353138 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:56:26.353425 6866 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:56:27.567282 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:56:27.567476 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:56:29.541918 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 51: this message will repeat every 5th retry.
W20250901 19:56:30.401585 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 51: this message will repeat every 5th retry.
I20250901 19:56:32.569093 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:56:32.569307 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:56:37.571076 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:56:37.571307 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:56:38.957228 7027 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:56:38.957441 7027 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:56:38.958236 7027 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:56:38.959390 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:56:38.960808 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:56:38.960978 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:56:38.961087 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:56:38.961362 7027 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250901 19:56:39.552341 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 56: this message will repeat every 5th retry.
W20250901 19:56:40.298564 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 56: this message will repeat every 5th retry.
I20250901 19:56:42.572896 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:56:42.573088 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:56:47.561329 5417 maintenance_manager.cc:419] P 08076b584d2a4aec8fd9868f85c9c7cd: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033354
I20250901 19:56:47.601004 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:56:47.601279 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:56:47.622577 5416 maintenance_manager.cc:643] P 08076b584d2a4aec8fd9868f85c9c7cd: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.060s user 0.030s sys 0.000s Metrics: {"bytes_written":15958,"cfile_init":1,"dirs.queue_time_us":443,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":1044,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":931,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":34,"thread_start_us":559,"threads_started":2,"wal-append.queue_time_us":399}
I20250901 19:56:47.623898 5417 maintenance_manager.cc:419] P 08076b584d2a4aec8fd9868f85c9c7cd: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1888 bytes on disk
I20250901 19:56:47.624681 5416 maintenance_manager.cc:643] P 08076b584d2a4aec8fd9868f85c9c7cd: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
W20250901 19:56:50.059695 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 61: this message will repeat every 5th retry.
W20250901 19:56:50.114205 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 61: this message will repeat every 5th retry.
I20250901 19:56:52.603034 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:56:52.603250 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:56:57.604980 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:56:57.605206 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:56:57.905758 7301 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:56:57.905943 7301 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:56:57.906790 7301 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:56:57.907749 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:56:57.909153 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:56:57.909353 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:56:57.909452 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:56:57.909744 7301 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250901 19:57:00.047292 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 66: this message will repeat every 5th retry.
W20250901 19:57:00.243912 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 66: this message will repeat every 5th retry.
I20250901 19:57:02.607156 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:57:02.607402 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:57:05.127768 7421 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:57:05.128014 7421 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:57:05.128777 7421 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:57:05.129978 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:57:05.131410 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:57:05.131594 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:57:05.131768 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:57:05.132082 7421 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:57:07.609192 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:57:07.609396 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:57:09.615762 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 71: this message will repeat every 5th retry.
W20250901 19:57:10.187052 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 71: this message will repeat every 5th retry.
I20250901 19:57:12.610836 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:57:12.611056 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:57:17.613308 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:57:17.613517 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:57:18.059521 3684 heartbeater.cc:646] Failed to heartbeat to 127.2.246.254:33523 (304 consecutive failures): Network error: Failed to ping master at 127.2.246.254:33523: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:57:18.214818 3829 heartbeater.cc:646] Failed to heartbeat to 127.2.246.253:34127 (318 consecutive failures): Network error: Failed to ping master at 127.2.246.253:34127: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:57:18.755838 3962 heartbeater.cc:646] Failed to heartbeat to 127.2.246.254:33523 (456 consecutive failures): Network error: Failed to ping master at 127.2.246.254:33523: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
I20250901 19:57:19.039362 7644 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:57:19.039585 7644 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:57:19.040355 7644 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:57:19.041450 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:57:19.043202 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:57:19.043452 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:57:19.043536 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:57:19.043808 7644 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250901 19:57:19.813804 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 76: this message will repeat every 5th retry.
W20250901 19:57:19.956146 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 76: this message will repeat every 5th retry.
I20250901 19:57:22.615123 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:57:22.615345 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:57:24.735615 7734 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:57:24.735862 7734 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:57:24.736517 7734 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:57:24.737751 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:57:24.739145 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:57:24.739375 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:57:24.739499 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:57:24.739857 7734 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:57:25.884974 7759 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:57:25.885190 7759 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:57:25.885840 7759 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:57:25.888191 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:57:25.888398 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:57:25.888505 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:57:25.888753 7759 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:57:27.616868 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:57:27.617095 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:57:29.651010 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 81: this message will repeat every 5th retry.
W20250901 19:57:30.032219 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 81: this message will repeat every 5th retry.
I20250901 19:57:32.618485 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:57:32.618688 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:57:37.620848 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:57:37.621112 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:57:39.584793 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 86: this message will repeat every 5th retry.
W20250901 19:57:39.835883 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 86: this message will repeat every 5th retry.
I20250901 19:57:40.841763 8009 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:57:40.841949 8009 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:57:40.842607 8009 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:57:40.843976 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250901 19:57:40.845381 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:57:40.845588 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:57:40.845690 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:57:40.845944 8009 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:57:42.622821 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:57:42.623054 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:57:47.624658 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:57:47.624869 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:57:49.835772 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 91: this message will repeat every 5th retry.
W20250901 19:57:50.013139 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 91: this message will repeat every 5th retry.
I20250901 19:57:51.396884 8187 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:57:51.397081 8187 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:57:51.397753 8187 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:57:51.398860 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:57:51.400591 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:57:51.400767 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:57:51.400873 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:57:51.401146 8187 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:57:52.626300 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:57:52.626489 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:57:56.388800 8268 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:57:56.389029 8268 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:57:56.389716 8268 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:57:56.392076 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:57:56.392267 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:57:56.392346 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:57:56.392596 8268 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:57:57.627974 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:57:57.628177 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:57:59.396183 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 96: this message will repeat every 5th retry.
W20250901 19:58:00.071612 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 96: this message will repeat every 5th retry.
I20250901 19:58:02.629734 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:58:02.629943 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:58:07.244333 8446 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:58:07.244640 8446 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:58:07.245370 8446 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:58:07.246718 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250901 19:58:07.247903 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:58:07.248093 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:58:07.248169 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:58:07.248394 8446 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:58:07.631052 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:58:07.631270 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:58:08.979506 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 101: this message will repeat every 5th retry.
W20250901 19:58:10.892802 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 101: this message will repeat every 5th retry.
I20250901 19:58:12.632635 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:58:12.632805 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:58:16.583750 8606 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:58:16.583961 8606 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:58:16.584724 8606 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:58:16.585856 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:58:16.588697 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:58:16.588874 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:58:16.588945 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:58:16.589198 8606 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:58:17.634397 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:58:17.634588 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:58:18.250726 3685 heartbeater.cc:646] Failed to heartbeat to 127.2.246.253:34127 (437 consecutive failures): Network error: Failed to ping master at 127.2.246.253:34127: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:58:18.428771 3829 heartbeater.cc:646] Failed to heartbeat to 127.2.246.253:34127 (438 consecutive failures): Network error: Failed to ping master at 127.2.246.253:34127: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:58:19.005110 3963 heartbeater.cc:646] Failed to heartbeat to 127.2.246.253:34127 (653 consecutive failures): Network error: Failed to ping master at 127.2.246.253:34127: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:58:19.005162 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 106: this message will repeat every 5th retry.
W20250901 19:58:20.637447 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 106: this message will repeat every 5th retry.
I20250901 19:58:22.635979 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:58:22.636160 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:58:27.637681 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:58:27.637897 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:58:28.731257 8800 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:58:28.731483 8800 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:58:28.732208 8800 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:58:28.733446 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:58:28.734724 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:58:28.734953 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:58:28.735033 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:58:28.735345 8800 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250901 19:58:28.850885 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 111: this message will repeat every 5th retry.
W20250901 19:58:30.454099 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 111: this message will repeat every 5th retry.
I20250901 19:58:32.639302 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:58:32.639488 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:58:37.640904 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:58:37.641095 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:58:39.178962 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 116: this message will repeat every 5th retry.
W20250901 19:58:40.998817 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 116: this message will repeat every 5th retry.
I20250901 19:58:42.642566 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:58:42.642772 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:58:47.644392 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:58:47.644600 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:58:48.392969 9116 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:58:48.393252 9116 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:58:48.394313 9116 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:58:48.395720 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:58:48.397328 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:58:48.397572 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:58:48.397672 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:58:48.397958 9116 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250901 19:58:49.031137 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 121: this message will repeat every 5th retry.
W20250901 19:58:50.335211 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 121: this message will repeat every 5th retry.
I20250901 19:58:52.646296 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:58:52.646492 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:58:57.647828 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:58:57.648015 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:58:58.762763 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 126: this message will repeat every 5th retry.
W20250901 19:58:59.505723 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 126: this message will repeat every 5th retry.
I20250901 19:59:01.742457 9332 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:59:01.742663 9332 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:59:01.743381 9332 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:59:01.744585 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:59:01.746626 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:59:01.746817 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:59:01.746902 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:59:01.747151 9332 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:59:02.649372 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:59:02.649569 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:59:07.651106 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:59:07.651329 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:59:08.003206 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 131: this message will repeat every 5th retry.
W20250901 19:59:09.700362 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 131: this message will repeat every 5th retry.
I20250901 19:59:12.653246 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:59:12.653451 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:59:14.726274 9548 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:59:14.726502 9548 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:59:14.727123 9548 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:59:14.728447 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:59:14.732653 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:59:14.732873 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:59:14.732966 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:59:14.733265 9548 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:59:17.654877 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:59:17.655087 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:59:17.797326 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 136: this message will repeat every 5th retry.
W20250901 19:59:18.446177 3685 heartbeater.cc:646] Failed to heartbeat to 127.2.246.253:34127 (557 consecutive failures): Network error: Failed to ping master at 127.2.246.253:34127: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:59:18.621770 3829 heartbeater.cc:646] Failed to heartbeat to 127.2.246.253:34127 (558 consecutive failures): Network error: Failed to ping master at 127.2.246.253:34127: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:59:19.064637 3963 heartbeater.cc:646] Failed to heartbeat to 127.2.246.253:34127 (839 consecutive failures): Network error: Failed to ping master at 127.2.246.253:34127: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:59:19.287385 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 136: this message will repeat every 5th retry.
I20250901 19:59:22.656540 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:59:22.656740 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:59:23.920696 9696 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:59:23.920923 9696 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:59:23.921612 9696 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:59:23.922794 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:59:23.924726 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:59:23.924971 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:59:23.925056 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:59:23.925333 9696 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250901 19:59:27.120535 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 141: this message will repeat every 5th retry.
I20250901 19:59:27.657996 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:59:27.658196 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:59:29.273873 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 141: this message will repeat every 5th retry.
I20250901 19:59:32.660002 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:59:32.660238 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:59:37.421326 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 146: this message will repeat every 5th retry.
I20250901 19:59:37.661851 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:59:37.662106 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:59:39.297489 3851 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer d397574faadb441192c99d530f14a734 (127.2.246.194:33557): Couldn't send request to peer d397574faadb441192c99d530f14a734. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 146: this message will repeat every 5th retry.
I20250901 19:59:40.681627 9963 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4c9163569ae1418090a245256e263186)
I20250901 19:59:40.681829 9963 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c9163569ae1418090a245256e263186" member_type: VOTER last_known_addr { host: "127.2.246.254" port: 33523 } } peers { permanent_uuid: "1d944ed54f8c43c9b8c4e7d589dad395" member_type: VOTER last_known_addr { host: "127.2.246.253" port: 34127 } } peers { permanent_uuid: "08076b584d2a4aec8fd9868f85c9c7cd" member_type: VOTER last_known_addr { host: "127.2.246.252" port: 37003 } }
I20250901 19:59:40.682581 9963 leader_election.cc:290] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4c9163569ae1418090a245256e263186 (127.2.246.254:33523), 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127)
W20250901 19:59:40.683970 5414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250901 19:59:40.685336 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4c9163569ae1418090a245256e263186 (127.2.246.254:33523): Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111)
W20250901 19:59:40.685535 5414 leader_election.cc:336] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1d944ed54f8c43c9b8c4e7d589dad395 (127.2.246.253:34127): Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
I20250901 19:59:40.685621 5414 leader_election.cc:304] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08076b584d2a4aec8fd9868f85c9c7cd; no voters: 1d944ed54f8c43c9b8c4e7d589dad395, 4c9163569ae1418090a245256e263186
I20250901 19:59:40.685827 9963 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 08076b584d2a4aec8fd9868f85c9c7cd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250901 19:59:42.663579 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:59:42.663818 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250901 19:59:47.141242 3849 consensus_peers.cc:489] T 6579c1fbcfc04dac9345a8157b352591 P e0064a1b18e5470b85bde2277757aac8 -> Peer c5b48d93671f4eb8980759b876e9ca21 (127.2.246.193:37767): Couldn't send request to peer c5b48d93671f4eb8980759b876e9ca21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6579c1fbcfc04dac9345a8157b352591. This is attempt 151: this message will repeat every 5th retry.
I20250901 19:59:47.665355 5395 master_runner.cc:186] Error getting master registration for 127.2.246.254:33523: Network error: Client connection negotiation failed: client connection to 127.2.246.254:33523: connect: Connection refused (error 111),
I20250901 19:59:47.665586 5395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 19:59:48.647724 5417 maintenance_manager.cc:419] P 08076b584d2a4aec8fd9868f85c9c7cd: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1888 bytes on disk
I20250901 19:59:48.653280 5416 maintenance_manager.cc:643] P 08076b584d2a4aec8fd9868f85c9c7cd: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.005s sys 0.000s Metrics: {"dirs.queue_time_us":369,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":122,"thread_start_us":540,"threads_started":2,"wal-append.queue_time_us":387}
W20250901 19:59:49.080142 3973 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:59:49.080461 3973 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:59:49.080562 3973 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
W20250901 19:59:49.080670 3973 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
F20250901 19:59:49.091526 3979 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8728509cfe374d5ea49110ea201c1659 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1756756789 (unix time) try "date -d @1756756789" if you are using GNU date ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 m*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095031 3983 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_788e14a6980547248f5627f671a238a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095031 3983 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_788e14a6980547248f5627f671a238a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095499 4002 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095031 3983 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_788e14a6980547248f5627f671a238a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095499 4002 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095928 3990 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095031 3983 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_788e14a6980547248f5627f671a238a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095499 4002 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095928 3990 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096103 3988 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_69070232fcfc4853b5fff18e23fbf965 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095031 3983 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_788e14a6980547248f5627f671a238a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095499 4002 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095928 3990 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096103 3988 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_69070232fcfc4853b5fff18e23fbf965 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096432 4003 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095031 3983 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_788e14a6980547248f5627f671a238a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095499 4002 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095928 3990 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096103 3988 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_69070232fcfc4853b5fff18e23fbf965 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096432 4003 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096561 4001 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095031 3983 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_788e14a6980547248f5627f671a238a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095499 4002 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095928 3990 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096103 3988 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_69070232fcfc4853b5fff18e23fbf965 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096432 4003 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096561 4001 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096655 3993 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095031 3983 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_788e14a6980547248f5627f671a238a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095499 4002 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095928 3990 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096103 3988 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_69070232fcfc4853b5fff18e23fbf965 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096432 4003 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096561 4001 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096655 3993 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097059 3982 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_14959779c0504cb4bf46faa666221361 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095031 3983 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_788e14a6980547248f5627f671a238a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095499 4002 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095928 3990 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096103 3988 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_69070232fcfc4853b5fff18e23fbf965 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096432 4003 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096561 4001 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096655 3993 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097059 3982 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_14959779c0504cb4bf46faa666221361 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097126 4000 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095031 3983 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_788e14a6980547248f5627f671a238a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095499 4002 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095928 3990 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096103 3988 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_69070232fcfc4853b5fff18e23fbf965 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096432 4003 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096561 4001 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096655 3993 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097059 3982 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_14959779c0504cb4bf46faa666221361 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097126 4000 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097236 3984 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6f60f64babef49459553b6e490e198e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095031 3983 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_788e14a6980547248f5627f671a238a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095499 4002 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095928 3990 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096103 3988 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_69070232fcfc4853b5fff18e23fbf965 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096432 4003 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096561 4001 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096655 3993 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097059 3982 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_14959779c0504cb4bf46faa666221361 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097126 4000 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097236 3984 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6f60f64babef49459553b6e490e198e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.098799 3985 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_edff33351bd64d16aec20db4230ac013 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095031 3983 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_788e14a6980547248f5627f671a238a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095499 4002 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095928 3990 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096103 3988 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_69070232fcfc4853b5fff18e23fbf965 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096432 4003 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096561 4001 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096655 3993 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097059 3982 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_14959779c0504cb4bf46faa666221361 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097126 4000 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097236 3984 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6f60f64babef49459553b6e490e198e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.098799 3985 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_edff33351bd64d16aec20db4230ac013 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.098801 3991 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000bdb) received by PID 3035 (TID 0x7f8f31db4700) from PID 3035; stack trace: ***
@ 0x7f8f44752980 (unknown) at ??:0
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095031 3983 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_788e14a6980547248f5627f671a238a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095499 4002 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095928 3990 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096103 3988 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_69070232fcfc4853b5fff18e23fbf965 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096432 4003 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096561 4001 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096655 3993 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097059 3982 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_14959779c0504cb4bf46faa666221361 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097126 4000 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097236 3984 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6f60f64babef49459553b6e490e198e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.098799 3985 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_edff33351bd64d16aec20db4230ac013 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.098801 3991 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.101154 3981 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b679bfff7a2f4d13b5e318b2ec42fc36 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250901 19:59:49.091619 3997 master-stress-test.cc:351] Check failed: _s.ok() Bad status: F20250901 19:59:49.091617 3986 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)Timed out: Error creating table default.Table_4c54782dd2bb4c66a3be449970979d0f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.092979 0000F03996000 master-stress-test.cc2025 328] 09 1 19:59Check failed: _s.ok() 49d status: 93020 3992 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.093012 3994 Timed out: AlterTable timed out after deadline expired: Alt: Table passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111) Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094154 3989 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094260 3999 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094312 3987 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7d788fb008945558d1e6fd4dde4f706 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.094753 3980 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_240cfd071a7c475bb44ff628d903bf9e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095031 3983 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_788e14a6980547248f5627f671a238a4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095499 4002 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.095928 3990 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096103 3988 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_69070232fcfc4853b5fff18e23fbf965 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096432 4003 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096561 4001 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.096655 3993 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097059 3982 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_14959779c0504cb4bf46faa666221361 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097126 4000 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.097236 3984 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6f60f64babef49459553b6e490e198e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.098799 3985 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_edff33351bd64d16aec20db4230ac013 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.098801 3991 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.101154 3981 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b679bfff7a2f4d13b5e318b2ec42fc36 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)F20250901 19:59:49.101177 3995 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.246.254:33523,127.2.246.253:34127,127.2.246.252:37003, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.246.253:34127: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f8f3d726fb7 gsignal at ??:0
@ 0x7f8f3d728921 abort at ??:0
@ 0x7f8f4045edcd google::LogMessage::Fail() at ??:0
@ 0x7f8f40462b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f8f4045e7cc google::LogMessage::Flush() at ??:0
@ 0x7f8f4045ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40b582 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:297
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7f8f3ddae6df (unknown) at ??:0
@ 0x7f8f447476db start_thread at ??:0
@ 0x7f8f3d80971f clone at ??:0